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Executive summary 


The project 


Research Learning Communities (RLC) was an intervention that aimed to improve teaching quality and 
learning outcomes by raising teachers’ awareness, understanding, and use of educational research in 
their teaching practice. Two Evidence Champion teachers from each school attended four RLC 
workshops with peers from up to four other schools. Workshops were delivered by academics from the 
Institute of Education at University College London and examined research relating to an area of interest 
agreed with the schools (for example improving literacy or numeracy via growth mindsets). The 
Evidence Champions were then required to develop, apply and evaluate school or key-stage wide 
improvement strategies using the learning from the workshops; and to support other teachers in the 
school, aiming to raise their awareness, understanding and use of research. 


The trial was funded by the Education Endowment Foundation (EEF), the Department for Education 
and the Mayor’s London Schools Excellence Fund as part of a round of funding exploring Research 
Use in Schools. The intervention ran for two academic years (from October 2014 to June 2016). This 
project used a randomised controlled trial to measure the impact of the intervention on teacher research 
engagement (measured with a survey) and on KS2 reading outcomes for Year 6 pupils. 


The evaluation was an efficacy trial. Sixty primary schools working across 14 RLCs were allocated to 
the treatment group, and 59 to the control group. A total of 5462 pupils were involved. The process 
evaluation included staff interviews, surveys of the Evidence Champions, and visits to RLC workshops. 


Key conclusions 


1. The project found no evidence that Research Learning Communities improves reading outcomes 
for children at Key Stage 2. 


2. The project did find a positive impact on teachers’ disposition towards research. There was, 
however, some evidence that this impact may have been influenced by other factors such as the 
level of postgraduate qualifications or seniority of teachers that took part in the intervention. 


3. Exploratory analysis identified some evidence of a small positive relationship between teachers’ 
disposition towards research and pupil outcomes, irrespective of involvement in an RLC. 


4. Evidence Champion roles in each school were intended to be held by the same people throughout 
the intervention, to support the development of a research-focused culture within each school. 
Staff turnover was therefore a barrier to implementation. 


5. Some teachers felt that it may take a number of years for participation in an RLC to change 
teaching practice and improve pupil outcomes. Future research could therefore examine longer 
term impacts. 


EEF security rating 


The finding for the primary outcome of reading has very high security. This trial was an efficacy trial, 
which tested whether the intervention worked under developer-led conditions in a number of schools. 
The trial was a well-powered, well-designed, two-armed randomised controlled trial. Relatively few 
pupils who started the trial were not included in the final analysis. The pupils in RLC schools were 
similar to those in the comparison schools in terms of levels of FSM eligibility and prior attainment. 


Additional findings 


There was no evidence that RLC had an impact on the primary outcome of reading at Key Stage 2. 
Impacts on the secondary outcomes of KS2 Numeracy and KS2 Grammar, Punctuation and Spelling, 
were generally small, and none were statistically significant. The results were similar for pupils eligible 
for free school meals, providing no evidence of a differential impact for these pupils. 
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The impact evaluation on teacher outcomes suggests that at the end of the intervention, teachers in 
intervention schools typically reported a greater disposition towards use of research evidence to inform 
their practice, in comparison to teachers in control schools, although once teacher background 
characteristics (such as leadership status and holding a postgraduate qualification) were controlled for 
this result was not statistically significant. 


There is also some evidence to suggest a link between the survey results and attainment outcomes. 
The more positive the disposition of KS2 teachers towards research use, the higher the pupils’ KS2 
Grammar, Punctuation, and Spelling outcomes, and KS2 Numeracy outcomes, irrespective of whether 
teachers were in intervention or control schools. 


The process evaluation found that staff turnover, and the competing priorities and limited time of 
teachers were barriers to successful implementation, because Evidence Champions need to be able to 
commit to the full duration of the intervention, attend all the workshops, and have time in school to 
develop their ideas and discuss the project with their colleagues. A lack of access to resources such as 
subscription journals may also limit the successful implementation of RLC activity by restricting the 
range and recentness of the evidence that teachers can draw on, although there is a general trend 
towards improving access to evidence for all audiences. 


The lack of impact on pupil outcomes discussed above may be partly due to the relatively short time 
scale: changes to school culture and teachers’ professional development may take longer than two 
years to translate into improved pupil outcomes. If this is the case, the tentative evidence of an impact 
on teachers’ research engagement, and of a relationship between teacher research engagement and 
pupil outcomes, indicates that it may be valuable to assess the impact of RLC over a longer time period. 


Cost 


The average cost for each school is around £860, or £3 per pupil per year when averaged over three 
years. The costs vary depending on the number of schools in an RLC, and the number of pupils in each 
school. In this intervention, there was also an additional one-off cost of £2500 per RLC for social network 
analysis, which was used to determine who would be best placed to act in the role of evidence 
champion, but this was not considered part of the intervention itself. 


Table 1: Executive summary table of impact on pupil outcomes 


Effect size Estimated EEF 
Outcome pie months’ P value Beco security 


of pupils | (95% confidence) —_ progress rating rating 


Readin 5462 0.02 0 month 0.74 cecee 
 Peadng | 82 | (-0.09,0.12) = 
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Introduction 


Intervention 


The Research Learning Communities (RLC) project focuses on promoting and embedding the use of 
research in schools. As part of a randomised controlled trial (RCT) involving 119 schools, 60 primary 
schools were allocated to the treatment condition for the intervention, delivered in 2014/2015 and 
2015/2016 by a team of academics from the Institute of Education at University College London. 


The delivery team used social network analysis to identify and recruit two to three potential ‘Evidence 
Champions’ from each school. This process allowed relationships between individuals in the school to 
be mapped and measured so that the nature and strength of those relationships could be understood. 
The process involved surveys with all teaching staff to determine which members of staff had most 
influence in terms of sharing ideas for practice. The intervention team recommended that at least one 
Evidence Champion came from the senior leadership team—to ensure that the intervention was seen 
by teachers to be supported by the school leadership—and the others could be any teacher to whom 
staff looked to for advice and ideas as highlighted by the social network analysis. Evidence Champions 
came together in RLCs. There were 14 RLCs in total and each comprised Evidence Champions from 
up to five schools. Each RLC met for a one-day workshop four times a year to examine research and 
evidence relating to an agreed area of focus. Prior to the first workshop, the intervention team offered 
schools a list of possible topics. Schools indicated from this list the options that they were interested in 
exploring, and a shared area which encompassed schools’ interests was suggested by the intervention 
team. Examples of foci included: 


e writing and the use of language; 


e consideration of how the use of feedback, marking, and moderation impact on the development 
of growth mindsets; 


e the role of teachers, parents, and pupils in feedback, marking and moderation; 


e effective grouping and setting and the development of growth mindsets amongst different 
groups of pupils; 


e growth mindsets; 
e improving literacy through growth mindsets; 


e improving literacy through the development of feedback, marking and moderation and the 
development of growth mindsets; 


e what makes outstanding teaching and learning in mathematics for all groups of pupils; 
e = effective feedback, marking and moderation; 

e improving literacy through feedback, marking and moderation; and 

e improving comprehension and use of language. 


The Evidence Champions took ideas from each workshop back to their schools to develop, apply, and 
evaluate school- or key-stage-wide improvement strategies based on this evidence. (For more details 
of activities in school, see ‘inter-sessional activities’ below.) The intention was for staff in schools to 
implement some kind of improvement strategy and collect data to understand its impact. Workshops 
were held in participating schools: some were held solely in one school (for example, if that school was 
particularly accessible and had good facilities for whole-day staff meetings); other RLCs alternated 
workshop venues over the course of the year. The evaluation focused on the cohort of pupils who were 
in Year 5 in 2014/2015, and Year 6 in 2016/2017. These pupils were due to undergo a statutory national 
assessment towards the end of the two-year intervention period, so there was no need to administer 
an additional pupil outcome measure to assess impact. 
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During the first year, a member of the delivery team who led the information-sharing in workshops 
facilitated group activities, supplied resources, and provided inter-sessional support for all participants, 
including a scheduled phone call and email contact with all individual Evidence Champions. The second 
year of the intervention involved RLCs being led by the Evidence Champions with ‘light touch’ support 
by the member of the delivery team: in this second year, the workshops were led by participants with 
the facilitator acting as a critical friend and guide, still providing inter-sessional support. In the second 
year, each workshop was divided into three parts: part one was led by participants and covered the 
main substantive focus detailed below, with the facilitator acting as a critical friend; part two was used 
to discuss with participants what was required from the following workshop, and who would be leading; 
part three included an element of capacity-building around research methods. All workshops referred 
to explicit learning outcomes as well as a recap of previous work. The workshop cycle agendas were 
the same in Year 1 and Year 2, the difference being in (a) the starting points of the schools and Evidence 
Champions, and (b) the responsibility for leading and facilitation. The plans below worked as a guide 
rather than an exact plan: in some cases, for example, discussion may have led to topics from a future 
workshop being introduced by the facilitator. 


The agenda of each workshop was as follows: 
Workshop 1 (November/December) 
e identifying and justifying what is good or best practice; 
e engaging with findings from research; 
e reflection on how research informs thinking about practice; 
e developing a research question for a school’s action research; 
e understanding baseline evidence; and 
e ways of sharing ideas with colleagues. 


Inter-sessional activities included creating formal opportunities for colleagues to engage with research, 
discussion of research question with colleagues, and collection of baseline data. 


Workshop 2 (January/February) 
e analysing baseline data; 
e considering current situation and potential ways forward; 
e discussing research on professional development; and 
e exploring ways of trialling a specific approach to practice. 


Inter-sessional activities included sharing the proposed approach with colleagues, and trialling the 
approach in school. 


Workshop 3 (March/April) 

e reflecting on how the approach is working, and how it can be refined; 

e engaging with evidence on change management; 

e consideration of plan for whole-school roll-out of approach; and 

e developing plan for how impact of whole-school roll-out can be measured. 
Inter-sessional activities include refining and rolling out approach, and recording impact. 
Workshop 4 (June/July) 

e analysis of impact data; 


e discussion of how knowledge about impact can be used by the school; 
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e determining the future focus of engagement with research; 
e accessing research evidence; and 
e understanding research quality. 


Some issues that arose with the delivery included: the continuity of the Evidence Champions across 
years (in some schools different people were allocated the role in Year 2); continuity of Evidence 
Champions across workshops (some schools sent different people to different workshops); and 
attendance at the workshops (not all evidence champions were able to attend every workshop, despite 
schools providing a commitment in the MoU to releasing evidence champions for the four workshops). 
These will be discussed in more detail later. 


Background evidence 


In recent years, there has been an international trend towards the use of evidence to inform practice 
(Caldwell et a/., 2015). Both in the U.K. and internationally, greater focus has been placed on addressing 
the gap between research evidence and professional practice and policy (Cooper, Levin and Campbell, 
2009; Nutley, Walter and Davies, 2007). In particular, policy makers have shown a growing interest in 
promoting evidence-based approaches in education (Caldwell et a/., 2015). For instance, initiatives 
such as ResearchED in the U.K. have aimed to bridge the gap between research and practice in 
education by bringing together researchers, teachers, and policy makers. These initiatives, in turn, have 
resulted in higher demands for research evidence (Bennet, 2015) and increased funding of projects 
that examine the impact of research-use in schools (Caldwell et a/., 2015). The fact that ‘existing 
teaching practice is not systematically evidence informed’ (Judkins, 2014, p. vi) is perhaps what has 
led to the emergence of these recent projects. 


Defining and agreeing on what the term ‘evidence’ represents still remains problematic (Caldwell et al., 
2015). Furthermore, there are many complexities in the relationship between research and practice in 
education (Caldwell et a/., 2015). To overcome these challenges, a number of researchers have 
attempted to define sharing of evidence and research as ‘knowledge mobilisation’ (Speight, Callanan, 
Griggs, and Cartagena Farias, 2016; Nelson, 2015; Levin, 2013; Cooper et al., 2009). Moreover, the 
recent development of a reliable research-use outcomes survey (by NFER and the EEF) is paving the 
way for defining and measuring teachers’ engagement with research, providing an instrument that can 
be used in a variety of contexts (Nelson, Mehta, Sharples and Davey, 2017). 


Nevertheless, Goldacre (2013) argues that despite much enthusiasm for, and the development of, a 
variety of programmes aimed to mobilise knowledge within the teaching profession, evidence-based 
approaches in teaching remain limited, and Brown and Zhang (2016) highlight that using evidence 
consistently to improve practice is, in many cases, difficult to achieve. He proposes that the lack of 
‘basic structures needed to support evidence-based practice’ as central to limited use of evidence- 
based approaches in teaching and suggests that the development of an ‘information architecture’ offers 
great promise (Goldacre, 2013, p. 15). This could include systems for disseminating research to 
teachers in easily accessible formats. Teachers also need to be aware of methods for conducting 
research to enable them to critically engage with research findings, and for some, potentially be able to 
conduct their own research. 


Sharples (2013) suggests that the ultimate goal should be to ‘empower professionals with evidence [...] 
integrating professional expertise with the best external evidence from research to improve the quality 
of practice’. To improve the effectiveness of research use in education, existing barriers need to be 
overcome (Cooper et a/., 2009; Levin, 2013). Skill and resource shortages (such as teachers’ inability 
to interpret research evidence, insufficient time, or lack of access to research literature) plus the lack of 
systemic incentives are some of the existing barriers to evidence- based practices in education (Speight 
et al., 2016; Nutley et a/., 2007; Sharples, 2013; Hemsley-Brown and Sharp, 2003). Furthermore, it is 
suggested that simply summarising research is not effective and that it is vital for evidence to be 
transformed for use by practitioners (Nelson ef a/., 2017). This, in turn, requires collaboration between 


Education Endowment Foundation 8 


Research Learning Communities 


researchers and teachers and moving beyond ‘knowledge exchange’ with the goal of effective 
implementation in schools (Speight ef a/., 2016; Nelson and O'Beirne, 2014). 


It is worth noting that there also remains an implicit assumption that a// research will have a positive 
impact for al/ teachers and schools (Caldwell et a/., 2015). While schools may have the capacity and 
the appetite to engage with research-based initiatives, there is no guarantee that this will result in 
improvements in educational outcomes (Nelson and O'Beirne, 2014; Caldwell et a/., 2015). In fact, a 
review by NFER found little evidence of impact on pupils’ attainment (Nelson and O'Beirne, 2014). This 
review highlighted the need to develop an infrastructure for knowledge mobilisation to support teachers 
in using evidence to develop their practice. ‘Knowledge mobilisation’ refers to the process through which 
evidence is produced through research, then made accessible to, and implemented by, practitioners. 
Nelson and O’Beirne highlighted the need for researchers to go beyond just summarising research for 
practitioners and outline how the findings can be used. They further suggested that there is a need for 
practitioners to develop skills in engaging with, and conducting, their own research to improve pupil 
outcomes. 


As highlighted by a number of reviews, until recently the literature on teachers’ knowledge and use of 
research evidence had been limited (Nelson and O'Beirne, 2014, Leat, Lofthouse and Reid, 2014; 
Dagenais et al/., 2012). While this body of research is rapidly growing, it is suggested there is still a gap 
in the literature on ‘the effects of evidence-based approaches on schools, teachers and pupils, and how 
to increase the likelihood of better outcomes for learners in particular’ (Caldwell et a/., 2015, p. 9). Asa 
result, more recently, projects have aimed to examine the relative efficacy of the different approaches 
to evidence-use, with a number of projects focusing on finding best approaches (Speight ef a/., 2016). 
There have been some promising findings from these recent projects. For instance, a survey study 
showed that while evidence-based approaches in education are still rather new, teachers recognise the 
potential of these approaches for informing their practice (Nelson et a/., 2017). Another study, 
addressing evidence-informed CPD, reported ‘some improvement’ in teachers’ perceptions of, and 
attitude to, evidence use, although this could not be attributed to the programme used in the study 
(Speight et al., 2016, p. 28). However, the teachers who were directly involved in the programme 
showed greater improvement, supporting the likely impact of the programme on their attitudes. Speight 
and colleagues (2016) identified lack of time and the difficulties associated with implementing the 
research in educational settings as the primary barriers to teachers’ use of evidence-based approaches. 


It can be argued that the use of randomised controlled trials to understand the impact of approaches to 
developing evidence-informed practice in education has inherent limitations. Caldwell ef al. (2015) 
highlight how this strategy simplifies the complexities of such approaches and their impact in different 
settings and contexts. Future research, therefore, needs to focus on identifying the relative 
effectiveness of different approaches and the conditions under which the benefits are realised (Caldwell 
et al., 2015), with the ultimate goal of developing ‘a range of tried and tested strategies for schools to 
adopt, in order to enable teachers to use research evidence to inform their practice’ (Judkins, 2014, p. 
6). 


The RLC intervention uses the concept of knowledge mobilisation to build on the emerging evidence- 
base exploring how teachers can use research to inform their practice. It aims to provide scope for 
teachers to engage with and use research evidence in ways that is appropriate for their own school 
cultures and own practice: in the intervention, teachers are facilitated and supported to develop their 
own research-informed cultures in schools. The intervention therefore acknowledges the complexity of 
contexts and seeks to support engagement with research within different contexts, rather than in spite 
of them. The evaluation took the form of a large-scale efficacy trial, with the developers delivering the 
intervention. The concept of Research Learning Communities is a new approach, a development from 
previous similar approaches such as Professional Learning Communities and Networked Learning 
Communities (Harris and Jones, 2012). This is the first time that Research Learning Communities has 
been run in schools. 
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Evaluation objectives 


The intervention was run by the developers across 59 schools: one school dropped out of the project 
after randomisation but before the intervention started. Research questions for the impact evaluation 
include: 


1. What is the impact of RLCs on: 
a. pupil attainment in Key Stage 2 reading (primary outcome)? 


b. pupil attainment in KS2 numeracy, grammar, punctuation and spelling (secondary 
outcomes)? and 


c. teachers’ awareness, understanding, and use of research, as measured by six 
variables on the research use survey? 


2. How do levels of teachers’ awareness, understanding, and use of research impact on pupil 
attainment in KS2 reading? 


Reading (formerly ‘literacy’) was chosen as a primary outcome by the EEF because it was seen to be 
the most general or broadest measure of pupil attainment available from KS2 tests. There was a wide 
range of foci over RLCs to ensure that schools were able to focus on what was relevant for their own 
contexts, so no single pupil outcome measure would directly match the foci for each RLC—hence the 
need for as general or broad a measure as possible. 


Research questions for the implementation and process evaluation include: 
1. Howdo RLCs and the role of Evidence Champions work in practice? 


a. What are the characteristics of schools where RLCs and the role of Evidence 
Champions appear to be successful? 


b. What barriers to the success of RLCs and the Evidence Champion role are reported 
by teachers, headteachers, and Evidence Champions? 


c. How attractive is the idea of Evidence Champions and RLCs to schools? 


d. What are the necessary prerequisites for the use of research to change teachers’ 
practice? 


e. How do teachers and Evidence Champions expect the process of RLCs to impact on 
their practice and on pupil attainment? 


2. How were the RLCs delivered? 


a. How does the Institute of Education work with Evidence Champions to develop their 
ideas about research? 


b. How do Evidence Champions enact their role in schools? 


3. What are the perceived outcomes of the Evidence Champions and RLCs in schools, including 
possible negative effects? 


4. Are there any ways in which RLCs and the role of Evidence Champions can be improved? 
5. How do control schools compare to RLC schools in their engagement with research evidence? 


The full evaluation protocol can be found here. 
Ethical review 


The evaluation underwent an ethical review by the University of Bristol’s Faculty of Social Sciences and 
Law Ethics Committee and was approved in July 2014. The project was conducted in accordance with 
the Graduate School of Education’s ethical practice guidelines and in line with British Educational 
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Research Association’s ‘Revised Ethical Guidelines for Educational Research’ (2004), ensuring 
anonymity. No individual school, teacher, or pupil are identifiable. Parents of the 2015/2016 Year 6 
cohort were given the opportunity to opt out of their child’s data being used for the research. Teachers 
from participating schools were informed via their schools by letter (prior to distribution of the survey 
link) about the nature of the research, and that they would receive a request to participate in the survey, 
but that they did not need to take part if they did not want to. Case study participants for the process 
evaluation were again informed about the nature of the research and that participation was voluntary. 


Project team 


The project delivery team was led by Dr Chris Brown at the Institute of Education, University College 
London. The intervention workshops were developed and delivered by members of the delivery team. 


The independent evaluation team was led by Dr Jo Rose at the University of Bristol, and supported by 
Professor Sally Thomas (impact evaluation lead), Professor Tim Jay, Dr Sara Speybroeck, Dr Anna 
Edwards, Dr Lei Zhang, Dr Andres Aguero, Ms Pooneh Roney, and Professor Kelvyn Jones. The team 
was responsible for the design and delivery of the evaluation, including collection of pupil outcomes 
from the National Pupil Database, baseline and outcome surveys of teachers in participating schools, 
process evaluation surveys of Evidence Champions, interviews with staff, observations of project 
workshops, and reporting of outcomes. Questionnaires for the baseline and outcomes surveys of 
teachers were developed by the National Foundation for Educational Research (NFER) (Poet et al., 
unpublished) and used across all projects in the ‘research use’ round. 


Trial registration 


The trial has been registered during reporting with registration number ISRCTN14382473. 


Education Endowment Foundation 11 


Research Learning Communities 


Methods 


Trial design 


The trial is an efficacy trial, with the intervention being delivered by the developers across treatment 
schools. The unit of randomisation is schools (or in some cases, clusters of schools) with a treatment 
or control condition. The control condition uses a ‘business as usual’ model, with schools being provided 
with a financial incentive of £1,000 for full participation in the evaluation. ‘Full participation’ involves all 
Year 5 and Year 6 teachers completing rounds two and three of the teacher survey; Year 3 and Year 4 
teachers also completing surveys—to give 75% of KS2 teachers overall completing rounds two and 
three of the teacher survey; and the provision of 2015/2016 Year 6 cohort UPNs in class lists with 
teacher names. Survey responses from Year 5 and Year 6 teachers were used in the second part of 
the analysis that matched teacher responses to pupil outcomes, and survey responses from teachers 
across the whole school were used in the teacher outcomes analysis comparing responses in control 
and treatment schools. Schools that did not ‘fully participate’ received a partial incentive: £500 for survey 
completion, or £250 for provision of UPN data. 


Participant selection 


The project sought to recruit primary schools in England; all primary schools were eligible regardless of 
location. The recruitment strategy was led by the delivery team. They approached all primary schools 
in the Institute of Education’s Research and Development Network (which allows members access to 
professional development opportunities and aims to support innovation and evaluation of new ways of 
working), and advertised the project through a range of other networks. Recruitment events were held 
in London, to which interested schools were invited to find out more about the project, and the delivery 
team also travelled to other areas of England including Devon, Leicestershire, and Yorkshire to discuss 
the project with other schools. The evaluation team attended some of the recruitment events to help 
explain what participation would entail, and also circulated the project details around their networks. 


The condition for being allocated to treatment or control groups was that at least 75% of KS2 teachers 
in the school complete the baseline (‘round one’) survey, and the headteacher and chair of governors 
of the school sign the Memorandum of Understanding. The latter included agreement to: participate in 
the evaluation following randomisation to the treatment or control condition; to support Evidence 
Champions’ participation in RLCs (if allocated to the treatment condition); to provide class lists and 
UPNs for pupils in the 2014/2015 Year 5 cohort and for pupils in the 2015/16 Year 6 cohort; and 
teachers completing rounds two and three of the surveys. Consent for participation in the intervention 
was therefore provided by the headteacher and chair of governors of each school, and consent for 
participation in the evaluation surveys and case-study interviews was provided by individual teachers. 
As KS2 attainment data is collected as standard practice in schools, parents were approached by 
schools in the 2015/2016 academic year for opt-out consent for the use of KS2 assessment data. The 
Memorandum of Understanding and consent forms can be found in Appendices C-F. 


Outcomes measures 


The primary outcome measure was 2015/2016 KS2 attainment data in reading for the cohort of pupils 
in Year 5 at the start of the intervention. Secondary outcomes include KS2 attainment in grammar, 
punctuation, spelling and numeracy, as well as six teacher outcomes as measured by the Research 
Use Survey (Poet et a/., unpublished; see Table 2 for names of individual items). The intervention aimed 
to impact on teachers’ practice through developing their engagement with research. The research foci 
of RLCs were determined by the Evidence Champions. Given that foci differed between RLCs, but the 
evaluation needed a single primary outcome (measure of pupil attainment) across the trial, the broadest 
measures of attainment available from KS2 testing were the most appropriate to use. Measuring 
teacher outcomes was crucial due to the nature and aims of the intervention. The final NFER Teacher 
Research Use Survey (round two and three) was developed for all EEF evaluations in the Research 
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Use round of interventions. While the original plan was to develop three measures of teacher 
awareness, understanding, and use of research, the final survey developed ended up with six measures 
of teachers’ engagement with research. Note that the initial baseline (round one) survey of teachers 
research use was different and much shorter than the subsequent final Teacher Research Use survey, 
although both were developed by NFER. The baseline survey was not intended to be used in any 
outcome analyses given it was designed only to provide basic snapshot information to the RLC 
intervention team as well as a means of demonstrating schools commitment to provide data and be 
involved in the project. 


KS2 attainment data were collected as part of the national testing system in England and data were 
obtained anonymously using UPNs and school identifiers from the National Pupil Database. A new 
format of KS2 testing was used in the 2015/16 academic year, with Literacy being split into two 
measures: Reading; and Grammar, Punctuation and Spelling. Writing is assessed via teacher 
assessment in the new KS2 tests but this measure was not employed due to concerns about low 
correlation with other KS2 measures (Allen, 2016) . Teacher surveys were collected via Bristol Online 
Surveys electronic survey tool. Table 2 shows the reliability of the six teacher measures created from 
the Teacher Research Use Survey in round two and round three (see Poet et a/., 2015 for further details 
of teacher measures). 


Table 2: Cronbach’s alpha scores for the survey measures 


Survey outcome measure Cronbach’s Cronbach’s Reliability 
Vl o)at-Matolelale W-mmmmmr-1) olat-Macele lace ie) 

Measure 1 (positive disposition to 0.84 0.84 Good 

academic research in informing teaching 

practice) 

Measure 2 (use of academic research to 0.62 0.59 Questionable 

inform selection of teaching approaches) 

Measure 3 (perception that academic 0.57 0.56 Poor 

research is not useful to teaching) 

Measure 4 (perception that own school 0.58 0.58 Poor 

does not encourage use of academic 

research) 

Measure 5 (active engagement with online 0.57 0.58 Poor 

evidence platforms) 

Measure 6 (Knowledge of research 0.53 0.52 Poor 


findings and methods) 


The reliability, in this project, of the survey measures used highlights the need for caution in the 
interpretation of any results using the survey data. Measures 3 to 6 have poor levels of internal reliability 
and as such the conclusions we can draw about these concepts are limited. 


Sample size 


Power calculations for this design were inevitably an approximation, with various assumptions being 
made about the structure of the data. A conservative calculation of the minimum detectable effect size 
was made using Optimal Design software, with the following assumptions: 


e Schools will be allocated in equal proportions to the treatment and control condition. 


e There will be a baseline covariate of KS1 reading and writing points scores, for KS2 literacy 
outcomes, at 0.73. 


e  Anintra-cluster correlation of 0.2. 
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e Where schools signed up to the project in groups, each group of two, three, four, or five schools 
will be treated as one unit (this makes the calculation particularly conservative as there will be 
a lot of variation within each of these units). In instances where individual schools have signed 
up not as part of a group, these individual schools will also be treated as a unit. (For further 
details of this process please refer to the section on randomisation below.) 


e An average of 50 pupils in the cohort per unit (this, again, is conservative as some units will 
comprise more than one school and will involve many more than 50 students—even though 
some schools will be smaller than this). 


With 90 units, the design would be able to detect an effect size on pupil attainment of 0.16. With 70 
units, the minimum detectable effect size rises to 0.18. 


The actual number of units used during the randomisation process was 71, giving a minimum detectable 
effect size of 0.18. 


Randomisation 


Schools were allocated to treatment and control groups using the minimisation method in MinimPy to 
ensure balance of school-level variables of size of school, %FSM, and attainment at KS2. The sample 
size was not large enough to warrant balancing on further variables as well. It was important to be 
aware of school alliances in this process. In total, 119 schools were recruited, though some signed up 
to the intervention in federations or alliances, and some as individual schools. Where groups of schools 
signed up, the size of the group determined how they were treated in the randomisation: 


e Large alliances of six schools or more (n = 5) were split into two groups using the minimisation 
method, based on size of school, percentage of FSM pupils, and KS2 attainment. Of these split 
alliances, one group was randomly allocated to the treatment condition and one to the control 
condition. 


e The remaining schools recruited were in units of either single schools (n = 54), federations or 
alliances comprising pairs (n = 6), or groups of three (n = 3) or four (n = 3) schools. 


This gave 71 units in total. These units were allocated to the treatment or control group using the 
minimisation method, based on size of unit (single school, two or three schools, and four or five 
schools), average size of school in each unit, average percentage of FSM pupils in each unit, and 
average KS2 attainment in each unit. 


The majority of units comprised single schools, but this method allowed for federations of schools or 
alliances who signed up because they wanted to work together. Using this method, 60 primary schools 
were allocated to the treatment group and 59 to the control group. 


Analysis of pupil outcomes 


There are two parts to the analysis of pupil attainment in 2016 KS2 outcome data (in both parts using 
the 2014/2015 Year 5 intention to treat cohort sample). These are subsequently referred to as ‘Part 1’ 
and ‘Part 2’. 


Part 1 uses multi-level modelling (MLM) to determine the effect size of the impact of the intervention on 
pupil attainment in reading at KS2 (primary outcome) and numeracy, grammar, punctuation and spelling 
(secondary outcomes)—accounting for clustering of pupils within schools (in random part by allowing 
random variation across schools). It was originally intended to consider ways to take account of 
clustering of intervention schools within RLC groups (see the Analysis section in the statistical analysis 
plan, 'SAP'), however, after testing preliminary models, this approach was not seen as appropriate due 
to confounding between RLC group and intervention and control categorisation. Moreover, grammar, 
punctuation, and spelling at KS2 is also used as a secondary outcome in addition to KS2 outcomes 
indicated in the SAP as this was considered to provide relevant and useful additional evidence on the 
impact of the RLC intervention on key aspects of literacy. Included in the fixed part of the model are: 
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e adummy variable for intervention group and prior KS1 attainment (from the NPD database); 


e FSMever eligibility dummy variable (from the PLASC/School Census), and relevant dummy 
variables for each of the variables used for the school randomisation (size of school in three 
categories (low/medium/high);' 


e %FSM in two categories (low/high);? and 
e 2013 KS2 attainment in two categories (high/low) as covariates.? 


Part 1 does not include, and is not linked to, the teacher survey data, but is a separate, stand-alone 
analysis. 


Part 2 involves linking individual teacher questionnaire data (from the 2015 round two survey) with pupil 
KS2 attainment data using combined Year 5 and Year 6 teacher name and student name UPN class 
lists provided by the schools (to match teachers to their pupils). For each pupil, six mean Year 5 and 
Year 6 Teacher Survey variables were calculated (reflecting different levels of teacher awareness, 
understanding, and use of research for the six measures described in Table 2 above) by averaging the 
round two survey responses of their specific Year 5 and Year 6 teachers. Multi-level modelling is used 
to understand whether different levels of teacher awareness, understanding, and use of research—as 
represented by these six variables—can explain, or are related to, differences in pupil attainment at 
KS2. The intervention dosage (number of RLC workshops attended by the Evidence Champion) is also 
examined in terms of whether this aspect can explain pupil KS2 outcomes as outlined in the SAP (see 
‘on-treatment analysis’ in Analysis section). The clustering of pupils within schools is taken into account 
using the same approach as in Part 1. The outcome variables for this are the same as Part 1 (2016 
KS2 attainment in the areas listed for the 2015 Year 5 intention-to-treat cohort sample). Round two of 
the survey will be used because teachers’ research use as indicated by the round two survey is more 
likely to be reflected in their teaching the following year leading up to the KS2 SATs assessment for the 
cohort of interest—and therefore may bear some relationship to pupil outcomes. Round three of the 
survey is collected six weeks or more after the SATs and would probably not be a good reflection of 
how teachers have taught over the previous two years leading up to the measurement of pupil 
outcomes. Changes that may have taken place in teachers’ research use between rounds two and 
three of the survey (as measured by the latter) would be unlikely to have fed through into their teaching 
practice, and from there into pupils’ learning, over the relevant timescale. 


Analysis of teacher outcomes 


The survey analysis uses multi-level modelling (MLM) to determine the effect size of the impact of the 
intervention on each of the six summary research engagement outcome measures, also taking account 
of the nesting of teachers within schools. The six teacher outcome measures are calculated for each 
teacher respondent using the round three teacher survey. Teacher type variables are explored as 
additional potential explanatory variable for use in the MLM analyses. Teacher type variables are: 


e status: 
o Classroom teacher; 
o middle leader (such as head of department, subject, or curriculum area leader); 
o senior leader (deputy or assistant headteacher); 


o headteacher, principal or director; 


1 ‘Low: 210 pupils or fewer (one class of 30 per year); ‘medium’: 211-420 pupils (one or two classes per year; 
control, N = 26; intervention, N = 25); ‘large’: over 420 pupils (more than two classes per year; control, N = 20; 
intervention, N = 17, where N is the school sample). 

2 ‘High’: above the 2013 national average of 19.2% for maintained nursery and primary schools (control, N = 26; 
intervention, N = 22); ‘low’: the 2013 national average or below (control, N = 31; intervention, N = 37). 

3 ‘High’: above the 2013 national average of 75% achieving level 4 in reading, writing and maths (control, N = 35; 
intervention, N = 44); ‘low: at the 2013 national average or below (control, N = 22; intervention, N = 15). 
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o SEN teacher; 
o supply, cover or Planning, Preparation and Assessment (PPA) teacher; 
o high level teaching assistant; 
o music or PE extra curricula teacher; 
e ‘holds postgraduate qualifications’; 
e years of teaching; and 
e years of teaching in current school. 


A separate analysis considers whether teacher responses to the survey change from round two, after 
the first year of the intervention, to round three, after the second year of the project at which time schools 
will be more independent in leading the RLCs. 


The analyses detailed above are conducted using Stata software (StataCorp, 2015) which provides 
equivalent results to other software such as MLwin. Due to the nature of the project and survey 
responses, it is impractical to conduct the analysis blind to group identity. 


On-treatment analysis 


In addition to the analysis to determine the effect size of the impact of the intervention on pupil KS2 
attainment, an additional, more sensitive, analysis alternatively uses the level of school engagement 
with the RLC programme instead of simply ‘intervention/control group’ in equivalent analyses of each 
student attainment outcome. This intervention dosage variable has been measured by the attendance 
records of Evidence Champions at eight workshops that took place over the two years of the project. 
Each workshop was either attended by no, one, or two Evidence Champions providing us with a variable 
ranging from O to 16 for each treatment school. This variable effectively describes the extent of 
intervention ‘dosage’ and provides additional evidence on the strength of the intervention impact. 


Subgroup analyses 


Separate subgroup analyses are also conducted for Part 1 of the student outcome analysis. This 
considers FSMever/non-FSMever groups and schools’ choice of focus in the workshops at the 
beginning of the project (literacy, 50 schools, or numeracy, 9 schools—as coded by the evaluation team 
according to information provided by the project team) as well as the interaction of these subgroups 
with intervention group. This analysis uses the final model to determine the effect size of the impact of 
the intervention on pupil attainment for subgroups. 


Effect size calculation 


The effect size is calculated in line with EEF guidance. This involves using the total variance after 
adjusting for prior attainment, FSMever, and other covariates described in Part 1 above. This is 
calculated using the random part estimate (pupil-level) from the final Multilevel model estimated in 
STATA. As indicated in SAP, all outcome and baseline attainment variables are standardised using 
normal scores as well as teacher Research Use Survey measures (1-6). The intervention dosage 
measure was not standardised but centred on mean. The effect size is represented by: (Coefficient for 
treatment group)/V(total variance after controls). The confidence interval is calculated using fixed part 
estimate for intervention dummy +/- 1.96 x Standard Deviation. 


All multilevel modelling analyses were conducted using Stata (StataCorp, 2015) and other analyses 
employed SPSS (IBM Corp, 2016). 


The statistical analysis plan (SAP) can be seen here. 
Implementation and process evaluation 


The process evaluation involved case studies of eight participating schools (six treatment and two 
control), observation of Evidence Champions at RLC workshops, and Evidence Champion surveys. 
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Case study schools were selected using purposive sampling, to allow for a range of approaches to 
using research, as defined by the first round of teacher surveys at the recruitment phase. Case studies 
and observations were carried out by members of the evaluation team, and surveys were distributed to 
schools by the evaluation team. 


It was planned to make two visits for each case study: one during the first year of the intervention cycle 
and one at the end of the second year. However, while all eight schools initially agreed to participate 
as case studies, one control school did not respond to repeated attempts to contact it in the second 
year, and one treatment school had a change of headteacher and, after initial discussion in the second 
year, did not respond to attempts to arrange a second visit. As such, six treatment schools and two 
control schools were visited in the first year of the intervention but only five treatment schools and one 
control school were visited in the second year of the evaluation. Each visit involved interviews with the 
Evidence Champions, a member of the senior management team, and, where available, at least one 
other Key Stage 2 teacher and the headteacher. The focus of the interviews was to understand school 
research cultures, teachers’ approaches to using and conducting research, their understanding of its 
effect on their practice, and reasons for their use (or non-use) of research in their practice. In addition, 
the interviews in treatment schools addressed the ways in which teachers and schools engaged with 
the RLC and how Evidence Champions experienced the RLC workshops and operationalised their role 
within the school. In total, 26 teachers from six treatment schools and 10 teachers from two control 
schools were interviewed in the first year of the evaluation, and 20 teachers from five treatment schools 
and four teachers from one control school were interviewed in the second year of the evaluation. 


Observations of RLC workshops (at least one from each of the eight rounds of workshops over the two 
years of the intervention) allowed us to understand Evidence Champions’ participation in the RLCs and 
the dynamics of the workshops. They also provided a context for some of the case-study schools. 


The survey of Evidence Champions included both open and closed questions about their engagement 
with research, their participation in the RLC workshops, their activities in school and with other schools 
as a result of the workshops, and their beliefs about the outcomes of the project. In total, 18 Evidence 
Champions from 14 schools responded to the survey in the first year of the evaluation, and 28 from 20 
schools responded in the second year. 


Fidelity was measured quantitatively by attendance at workshops (see ‘on-treatment analysis’ in the 
impact evaluation). This was collected by the delivery team at each workshop. Because of the large 
scale of the trial—and the range of different ways in which Evidence Champions might work to develop 
a research-oriented culture in their own school—it was not appropriate to use activities outside of RLC 
workshops as a measure of fidelity. Instead, observations of one or two of each round of workshops, 
case-study interviews, and Evidence Champion surveys aimed to understand the range of ways in 
which the delivery team worked with Evidence Champions, and Evidence Champions worked in their 
schools. 


Costs 


The cost of delivering the intervention to each RLC was provided by the delivery organisation. The 
evaluation team calculated the cost at £3.14 per child per year for a 3-year intervention. 


Information about the amount of time spent in schools by Evidence Champions and other teachers 
engaging in research were collected through the process evaluation and discussion with the delivery 
team—although this varied very widely between schools including some schools that did not allocate 
any extra dedicated time beyond the workshops. 
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Table 3: Timeline of the project 


June 2014— 


Activity 


Nationwide recruitment of participating schools by the intervention team, 


September 2014 


June 2014— 
September 2014 


July 2014— 
September 2014 


September 2014 
October 2014 


November 2014 
January 2015 


March 2015—April 
2015 


March 2015—May 
2015 


June 2015 


June 2015—October 
2015 


September 2015- 
July 2016 


November 2015 
January 2016 


February 2016-— 
March 2016 


March 2016-April 
2016 


May 2016- 
September 2016 


June 2016 


June 2016—October 
2016 


July 2016- 
September 2016 


September 2016 


with support from the evaluation team at recruitment events. 


Teacher baseline survey administration (as part of recruitment process) 
using baseline research-use questionnaire developed by NFER. 


Collection of UPNs for 2014/2015 Year 5 cohort in participating schools, in 
class lists with teacher names. 


Allocation of schools to treatment and control groups, using minimisation. 


Evidence Champions suggested to schools by the project team, according to 
social network analysis for each school’s staff team. 


Workshop 1, Year 1. 
Workshop 2, Year 1. 
Workshop 3, Year 1. 


Case study visits for process evaluation. 


Workshop 4, Year 1. 


Teacher Survey round two administration using Research Use 
Questionnaire developed by NFER. Evidence Champion survey 
administration. Links were sent to schools in June 2015, with reminders by 
email and telephone in July, September, and October 2015. (The deadline 
was extended due to low response rates.) 


Collection of UPIs for 2015/2016 Year 6 cohort in participating schools, in 
class lists with teacher names. 


Workshop 1, Year 2. 
Workshop 2, Year 2. 


Parent opt-out consent letters sent to schools for distribution to parents. 


Workshop 3, Year 2. 


Case study visits for process evaluation. 


Workshop 4, Year 2. 


Teacher Survey round three administration using Research Use 
Questionnaire developed by NFER. Evidence Champion survey 
administration. Links were sent to schools in June 2016, with reminders by 
email and telephone in July, September, and October 2016. (As in round 
two, the deadline was extended due to low response rates.) 


Liaising with NPD to develop application for data. 


Application for NPD data. 
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October 2016— Cleaning of teacher survey data. 
December 2016 Development of statistical analysis plan. 
Analysis of process evaluation data. 


January 2017—May Receipt of NPD data. 
2017 Matching of NPD data to teacher survey data. 


Analysis of NPD data and teacher survey data. 
Analysis of process evaluation data. 
Synthesis of findings and report writing. 
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Participants 


Figure 1: Participant flow diagram for primary outcome (KS2 reading) 


Approached (school n > 200) 


Did not agree to participate 
(school n > 40) 


= 
oO 
= Agreed to participate (school 
S n= 161) Excluded (school n = 42) 
oc Not meeting inclusion criteria 
(school n = 42) 
Other (school n = 0) 
Randomised 
(school n = 119; pupil n = 5,462) 
6 
@ Treatment Waitlist control 
& (school n = 60; (school n = 59; 
<x pupil n = 2,619) pupil n = 2,843) 
“note one school with 
no KS2 pupils was 
randomised by error, 
pupil numbers were not 
included in this total. 
Lost to Post-test Lost to Post-test 
follow up data follow up data 
(Missing collected (Missing collected 
Z Pupil n = 7) 59; Pupil n = 19) 57; Pupil n 
x) (Other Pupil n = (Other = 2,740 
ie schools: 2,546 schools: pupil 
pupil n = 66) n = 84) 
Not analysed Analysed Not analysed Analysed 
» | Pupiln = 144 Pupil n = 2,402 Pupil n = 176 Pupil n = 2,564 
@ (other missing (other missing 
| datae.g. KS1 data e.g. KS1 
& or FSM) or FSM) 
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Table 4: Minimum detectable effect size for primary outcome of pupil attainment at different 
stages 


Correlation 
Ni EXeq aTeXe) Fy between Blocking/ 
(n= pre-test stratification 
interwontion'n (+other (ele; or pair Power Alpha MDES 
EMerel ani ice))) covariates) matching 
& post-test 
110 (55; 55) Did not use 0.16 
Protocol OR 0.73 0.2 in 80% 0.05 OR 
70 Units calculations 0.18 
119 (60; 59) Did not use 
Randomisation OR 0.73 0.2 in 80% 0.05 0.18 
71 Units calculations 
116 (59; 57) Did not use 
Analysis OR 0.63 0.11 in 80% 0.05 0.17 
70 UNITS calculations 


While Figure 1 details the participant flow for the primary outcome, the details for the teacher outcomes 
and Part 2 of the pupil outcomes analysis (matching pupil outcomes to teacher outcomes) are presented 
below. 


Treatment Schools 
e Discontinued intervention at the start or partway through the project: school n = 4. 


e Of the remaining schools (n = 56), 430 teachers in 54 schools completed the survey in round 
two, and 227 teachers in 39 schools completed the survey from round three. 


e For teacher survey outcomes: not analysed, school, n = 21; teacher, n = 802; analysed: school, 
n = 39; teacher, n = 235. 


e Received pupil UPNs: school, n = 55; pupil n = 2,599. This enabled us to access and match 
KS2 attainment data and teacher survey data for pupils for Part 2 of KS2 outcome analysis. 


e Data on attendance at workshops is available for n = 43 schools. Other teacher attendance 
data is missing as this was not collected by UCL workshop leaders. 


Conirol Schools 
e Discontinued participation partway through the project: school n = 2. 


e Of remaining schools (n = 57), 528 teachers in 49 schools completed the survey in round two, 
and 468 teachers in 41 schools completed the survey in round three. 


e For teacher survey outcomes: not analysed, school, n = 19; teacher, n = 638; analysed: school, 
n = 40; teacher, n = 464. 


e Received pupil UPNs: school, n= 53; pupil, n = 2,829. This enabled us to access and match 
KS2 attainment data and teacher survey data for pupils for Part 2 of KS2 outcome analysis. 
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Sample characteristics 


Table 5 presents the comparison of school characteristics at baseline, and Tables G.5—G.7 in Appendix 
G present this comparison for those in the different analyses. Percentages are rounded to the nearest 
whole number so may not always sum to 100. Size of school, percentage of FSM eligibility, attainment 
at KS2, location, and school type are all taken at time of baseline. Percentages of pupils with EAL, with 
SEN, and who are boys, Ofsted ratings, and levels of FSM eligibility, are all from available 2016/2017 
data for the schools. To check balance at baseline, t-tests were conducted on size of school, %FSM, 
%KS2 level 4, SEAL, %SEN, and %boys. None of the t-tests showed a significant difference between 
treatment and control groups: 


e size ofschool: t(117) = 0.739, p = 0.461, hedges g = 0.14 
e %FSM: t(117) = 0.443, p = 0.659, hedges g = 0.08 
e %KS2 Level 4: t(117) = -1.589, p = 0.115, hedges g = 0.22 
e = %EAL: t(117) = -0.289, p = 0.773, hedges g = 0.05 
e %SEN: (117) = -0.013, p = 0.990, hedges g = 0.002 
e %boys: t(117) = 0.711, p = 0.478, hedges g = 0.13 


A comparison of pupil-level standardised KS1 results at baseline for Part 1 analysis showed no 
significant difference, with a negligible effect size: 


e pupil-level KS1: t(5102) = 0.153, p = 0.878, hedges g = 0.043. 


Missing data for the KS2 outcomes analysis (Part 1) is minimal because this data comes entirely from 
Year 5 intention to treat NPD records. Part 1 analysis is based on the full cohort of Year 5 KS2 students 
in the NPD for participating schools. NPD data was unavailable for three schools in the sample: two 
control schools and one treatment school. For two very small schools, the reason for the missing KS2 
data is unclear. In the case of the third school, an infant school with no KS2 students was included by 
error in the original UCL list of potential schools to participate in the RLC project and was selected in 
the randomisation procedure as a control school, although no pupil numbers were included as 
randomised or as attrition due to the error (see Appendix G). 


For the KS2 outcomes analysis, Part 2, the main source of missing data relates to missing Year 5 pupil 
UPNs and school engagement (measured through attendance at workshops) which cannot be 
imputed). There is also some missing teacher survey data (round two) and intervention dosage (number 
of workshops attended by evidence champion). Otherwise there is no significant missing data on key 
analysis variables (KS1 and KS2 pupil attainment, and FSM eligibility). Overall data was missing for 
between 15-37 schools for these analyses: 10-24 control schools and 5—13 intervention schools (see 
Appendix G). 


For the teacher outcomes analysis, there is some missing data on teacher survey data (round three). 
For those teachers responding to the survey, there was no missing item data and thus no effect on the 
calculation of the six summary measures. Overall, data was missing for the teacher survey (round two) 
for 11 schools: seven control schools and four intervention schools, and for teacher survey (round three) 
for 40 schools: 19 control schools and 21 intervention schools (see Appendix G). 


School characteristics for participating schools at different stages of the analysis can be seen in Table 
G.5-G.7 in Appendix G. In order to check the sensitivity of the findings to the extent of missing data 
(the potential impact of missing values on the comparison between intervention and control schools) 
the balance at different stages of analysis was also examined. T-tests were conducted on size of school, 
%FSM, %KS2 level 4, %EAL, %SEN, and %boys. None of the t-tests showed a significant difference 
between intervention and control groups, except that there is a marginally significant difference between 
control and treatment schools for Part 1 analysis in terms of school-level KS2 attainment (% achieving 
level 4 at KS2) at baseline, with a small effect size of 0.37, and there is a significant difference between 
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control and treatment schools for Teacher Survey round three analysis in terms of size of school, with 
a medium effect size of 0.55. These findings indicate that, overall, in spite of reductions in the sample 
size for different analyses, the balance between intervention and control school in a range of school 
characteristics remained almost entirely consistent. 


This sample has larger schools than the national average for primary schools (279). Also, compared to 
national averages, a higher percentage of FSM pupils (14.1%), EAL pupils (20.6%), and pupils reaching 
level 4 in KS2 reading, writing and mathematics (53%). (Average figures, DfE, 2016, 2017.) 
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Variable Taiccyarc-valicelame | gel] e) Ofeyaiice) me] cele] e) 
School-level (continuous) n/N Mean n/N Mean 

(missing) (missing) 
Size of school (number of 60/60 (0) 319.2 59/59 (0) 341.1 
pupils) 
% FSM eligibility 60/60 (0) 20.7% 59/59 (0) 22.0% 
Attainment at KS2 60/60 (0) 79.3% 59/59 (0) 75.8% 
(% achieving level 4 in 
reading, writing and maths) 
% Pupils with English as an 60/60 (0) 25.9% 59/59 (0) 24.5% 
Additional Language (EAL) 
% Pupils with SEN 60/60 (0) 3.2% 59/59 (0) 3.2% 
% Boys 60/60 (0) 50.9% 59/59 (0) 51.6% 
Pupil-level attainment at 2,454/2,454 0.0100 2,650/2,650 0.0143 
KS1 (standardised) (0) (0) 

n/N n/N 

School-level (categorical) (missing) Percentage (missing) Percentage 
Ofsted Rating 46/60 (14) 77% 46/59 (13) 78% 
Outstanding 9/60 (0) 15% 11/59 (0) 19% 
Good 36/60 (0) 60% 33/59 (0) 56% 
Requires Improvement 1/60 (0) 2% 2/59 (0) 3% 
Inadequate 0/60 (0) 0% 0/59 (0) 0% 
None 14/60 (0) 23% 13/59 (0) 22% 
School Location 
Urban 50/60 (0) 83% 52/59 (0) 88% 
Rural 10/60 (0) 17% 7/59 (0) 12% 
FSM eligibility 
High (> 35%) 13/60 (0) 22% 11/59 (0) 19% 
Medium (20.1%-35%) 8/60 (0) 13% 14/59 (0) 24% 
Low (< 20.1%) 39/60 (0) 65% 34/59 (0) 58% 
School Type 
Academy—Converter 9/60 (0) 15% 5/59 (0) 8% 
Mainstream 
Academy—Sponsor Led 5/60 (0) 8% 9/59 (0) 15% 
Mainstream 
Community 33/60 (0) 55% 30/59 (0) 51% 
Community Special 0/60 (0) 0% 1/59 (0) 2% 
Foundation 1/60 (0) 2% 1/59 (0) 2% 
Foundation Special 1/60 (0) 2% 0/59 (0) 0% 
Voluntary Aided 10/60 (0) 17% 8/59 (0) 14% 
Voluntary Controlled 1/60 (0) 2% 5/59 (0) 8% 
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Outcomes and analysis 


Summary tables of the multilevel modelling results are presented below and the specific model results 
are presented in Appendix H. 


Student outcomes Part 1 analyses 


Tables 6 and 7 show the overall impact on pupils’ KS2 attainment of the RLC intervention. The basic 
unit of analysis is pupils (clustered within schools). It was found that there is no statistically significant 
difference (the effect size is low, at < 0.2) between intervention and control schools in KS2 pupil 
outcomes (primary and secondary), either for all pupils or for pupil subgroups (FSM, RLCs with literacy 
focus, RLCs with numeracy focus). 


Table 6: Primary outcome analysis 


Raw means 
Intervention 
group Ofoyaiice) mel gelele) 
ninmodel Effect size 
Outcome n Mean n Me : : A p- 
, ee Pe (Tah Yayoi t(o) alm (Oxo) ay ice (=Valer=) 
(standardised) (missing) (95% Cl) (missing) (95% Cl) - control) Interval) value 
0.02 -0.02 4,966 
KS2 reading ane (-0.02, aa (-0.06, (2,402; ee 0.12) 0.74 
0.06) 0.01) 2564) laa 
-0.29 -0.28 
KS2 reading (FSM 898 (-0.35 969 (-0.34 1724 0.04 0.52 
upils ls ane : : : 
pupils) (40) 0.23) (40) 0.22) (822;902) — (-0.06, 0.12) 
KS2 reading (schools -0.02 
with a literacy focus to 2,257 rae 2,740 -0 06 ee 0.09 0.68 
the intervention in Y1 | (69) 0 11) (103) i 564) (-0.07,0.20) ~~ 
vs control) : 0.01) 564) 
KS2 reading (schools 
with a numeracy 289 0.30 Jo740 | 092 | a aga -0.01 
focus to the 4 (-0.43, 103 (-0.06, 69: 2.564 0.24. 0.08 0.91 
intervention in Y1 vs (4) -0.17) (103) 0.01) (269; 2,564) | (-0.24, 0.08) 
control) 


Note: The effect sizes were estimated after controlling for prior attainment and everFSM, intervention and 
randomisation control variables. 
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Table 7: Secondary outcome analysis 


Raw means 
Intervention group (Ofoy alice) Me [cell] e) 
am lamarcete(=)| 

Outcome Re Mean n Mean . ‘Effect 
(standardised) " (™ISSiNG) (950, Gl) (missing) |(95% Cl) wie a, eize(Cli we 
KS2 grammar, 0.03 -0.03 4,997 0.10 
punctuation and an (0.01, ys (0.07, (2,412; (-0.06, | 0.27 
spelling 0.07) 0.01) 2,585) 0.20) 
KS2 grammar, 024 -0.30 0.09 
punctuation and 866 (-0.28 941 (-0.36 1742 (-0.02 0.14 
spelling (FSM 32 ek 28 oe. 829; 913 a , 
pupils) Se) -0.15) 28) -0.23) ( ) 0.14) 
KS2 grammar, 
spelling (schools 2,268 | (0.03 209. | (0.07 (2.142: (0.07, 0.35 
with a literacy focus (58) a (78) 0.01 , 9 G5) Seite ; 
to the intervention 0.11) 01) 585) 0.20) 
in Y1) 
KS2 grammar, 
punctuation and -0.24 -0.03 0.15 
spelling schools 292 (-0.37 2,765 (0.07 2,855 (0.15, 0.41 
with a numeracy (1) : (78) aa .| (200; 2585) aa , 
focus to the -0.11) 0.01) 0.36) 
intervention in Y1) 

0.01 -0.01 0.06 
KS2 numeracy a (-0.03, pet (-0.05, 5 eee (-0.08, 0.54 

0.05) 0.03) at, 0.16) 
Ree nlmetaey Bee ne aa ee ee her 0.68 
(FSM pupils) (32) 0.19) (24) ‘01 7) (830; 915) 0.07) ‘ 
KS2 numeracy : 
croomwins aaa G8 ary (201 re 008 
literacy focus to the (58) ae (72) i ¢ 588) 2 , 
intervention in Y1) 0.08) 0.03) 588) 0.16) 
KS2 numeracy 
(schools with a 992 -0.23 2771 -0.01 2 858 0.12 
numeracy focus to { (-0.37, 79 (-0.05, 970. 9 588 (-0.15, | 0.46 
Oil in (1) -0.10) (72) 0.03) (270; 2,588) 0.34) 


Note: The effect sizes were estimated after controlling for prior attainment and everFSM, intervention and 
randomisation control variables. 
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Student outcomes Part 2 analyses 


Tables 8 and 9 show more sensitive analyses of the impact on pupils KS2 attainment of the RLC 
intervention dosage and KS2 (Year 5/6) teachers’ research use to inform their practice (measures 1 
and 2, Teacher Survey round two) as outlined in the SAP. The basic unit of analysis is pupils (clustered 
within schools). KS2 pupil records were matched via UPNs provided by schools to the teacher (round 
two) survey results of their KS2 (Year 5/6) teacher. Due to missing UPN and teacher survey data this 
resulted in a smaller sample size than Part 1 analyses (see Appendix G). The results for KS2 Teacher 
Survey (round two), measures 3-6, were not found to be statistically significant for any KS2 outcome 
but are presented for information in Appendix H (Tables H.6—-H.8). Overall, it was found that: 


For KS2 reading there is no statistically significant impact of RLC intervention dosage (effect 
size = 0.01) or Teacher Research Use Survey (round two) measure 1 (effect size = -0.003) and 
measure 2 (effect size = 0.06). 


For KS2 grammar, punctuation, and spelling there is no statistically significant impact of RLC 
dosage (effect size = 0.02). However, there is a statistically significant impact of Teacher 
Research Use Survey (round two) measure 1 (effect size = 0.07) and measure 2 (effect size = 
0.09). This suggests the more KS2 (Year 5/6) teachers report using research evidence to inform 
their practice, the higher are their pupils’ KS2 grammar, punctuation and spelling scores, 
irrespective of whether they teach in an intervention or control school. 


For KS2 numeracy there is no statistically significant impact of RLC intervention dosage (effect 
size = 0.01) or Teacher Research Use survey (round two) measures 1 (effect size = 0.05). 
However, there is a statistically significant impact of Teacher Research Use survey (round two) 
measure 2 (effect size = 0.08). This suggests the more KS2 (Year 5/6) teachers report using 
research evidence to inform their practice, the higher their pupils KS2 numeracy score, 
irrespective of whether they teach in an intervention or control school. 
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Table 8: Impact of Teacher Survey (round two) research use in school, measure 1 and 2 
analysis on KS2 outcomes 


Teacher research use in 


Effect size 


Pupil Outcome eee N in model (95% Cl) P-value 
Measure 1 (positive 
disposition to academic -0.003 
research in informing aor! (-0.05, 0.04) one 
KS2 reading (primary teaching practice) 
puleonte) Measure 2 (use of 
academic research to 0.06 
inform selection of een (-0.01, 0.08) as 
teaching approaches) 
Measure 1 (positive 
disposition to academic 0.07 
research in informing oree (0.01, 0.09) ce 
KS2 grammar, teaching practice) 
punctuation, and 
spelling Measure 2 (use of 
academic research to 0.09 
inform selection of enee (0.01, 0.10) 0.01 
teaching approaches) 
Measure 1 (positive 
disposition to academic 0.05 
research in informing 2,886 (0.002, 0.08) O07 
teaching practice) 
KS2 numeracy 
Measure 2 (use of 
academic research to 0.08 
inform selection of a6ee (0.01, 011) oo! 
teaching approaches) 


Note: The effect sizes of measures 1 and 2 were estimated after controlling for prior attainment and everFSM. 
Randomisation control variables were not included in this analysis as it was not required given intervention dummy 
variable was not included (due to overlap/confounding with teacher survey research use variables). 
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Table 9: Impact of on-treatment (intervention dosage) analysis on KS2 outcomes 


Pupil outcome 


Intervention 


IN Tam earexe(=)| 


TUT atl ey=) ae) i 


Effect size 


P-value 


fo foy-¥- 10 [=] (intervention; schools in (95% Cl) 
Corey al ige))) rnateye (=) 
(intervention; 
(exe) a} ine) )) 
; intervention dosage 
ie reading (number of RLC 4,131 (2,025; 104 0.01 a 
4 y workshops 2,106) (55; 49) (-0.00, 0.01) , 
outcome) attended) 
intervention dosage 
irri ata (number of RLC 4,153 104 0.02 ar 
P patel workshops (2,033; 2,120) (55; 49) (-0.01, 0.02) ‘ 
and spelling attended) 
intervention dosage 
KS2 numeracy | ("umber of RLC ace oe ot 0.15 
Y workshops (2,033; 2,120) (55; 49) (-0.00, 0.02) 
attended) 


Note: the effect sizes of intervention dosage were estimated after controlling for prior attainment, everFSM, and 
randomisation control variables. 


Table 10 shows the overall impact of the RLC intervention on all KS2/KS2 teacher outcomes evaluated 
in terms of six measures created from the Teacher Research Use Survey at the end of the intervention 
(round three). The basic unit of analysis is KS2 /KS2 teachers (clustered with schools). It was found 
that there is a statistically significant difference between intervention and control schools in two teacher 
outcomes (measures 1 and 2) but not in the other four measures (measures 3-6), although analysis of 
the difference between intervention and control schools approaches statistical significance. The effect 
sizes were 0.21 and 0.20 for measures 1 and 2, respectively and between 0.12 and -0.17 for the other 
measures, 3-6. This indicates that at the end of the intervention (round three), teachers in intervention 
schools typically report a higher use of research evidence to inform their practice, in comparison to 
teachers in control schools. (See also ANOVA results below.) 
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Table 10: Teacher outcome (round three) analysis 


I Tam earexe(=)| 


BI-Y-(et-)molUl Coxeliil-M(gelelave Mm tale-y:)) (intervention; anes me s P-value 
control) Ca) 

Measure 1 (positive disposition to 699 0.21 

academic research in informing ; . 0.03 

teaching practice) (235; 464) (0.02, 0.39) 

Measure 2 (use of academic research 699 0.20 

to inform selection of teaching . . 0.05 

approaches) (235; 464) (0.00, 0.39) 

Measure 3 (perception that academic 699 -0.17 0.06 

research is not useful to teaching) (235; 464) (-0.35, 0.01) , 

Measure 4 (perception that own 699 0.11 

school does not encourage use of . . 0.32 

academic research) (235; 464) (-0.32, 0.10) 

Measure 5 (active engagement with 699 0.12 0.18 

online evidence platforms) (235; 464) (-0.12, 0.23) ; 

Measure 6 (knowledge of research 699 0.005 0.96 

findings and methods) (235; 464) (-0.18, 0.19) , 


Note: The effect sizes were estimated after controlling for intervention and randomisation control variables. 


Further analyses were also conducted to test the association of teacher background variables from the 
teacher survey to teachers’ research use outcome variables, having already controlled for 
intervention/control group and randomisation variables. (The teacher background variables are detailed 
on page 15.) The cross-tabulation tables of these variables by intervention/control can be seen in Table 
G7.8-—G7.11 in Appendix G. This approach was helpful to explore whether certain teacher 
characteristics also play a role promoting teachers research use, over and above the RLC intervention. 
The findings showed a statistically significant association of some categories of teacher status (in 
comparison to classroom teacher) and more positive reports of teachers research use. Irrespective of 
whether in an RLC intervention or control school: 


e headteachers responded more positively in all six measures; 


e senior leaders responded more positively in all measures except measure 2 (use of academic 
research to inform selection of teaching approaches) and measure 4 (perception that own 
school does not encourage use of academic research); and 


e middle leaders responded more positively in all measures except measure 4 and measure 6 
(knowledge of research findings and methods). 


Moreover, irrespective of whether in an intervention or control school, teachers holding a postgraduate 
qualification also responded statistically significantly more positively on measure 1 (positive disposition 
to academic research in informing teaching practice) and measure 6. Finally, years of teaching in 
current school was found to have a small statistically significant association: teachers who had taught 
longer in their current school were more likely to respond negatively on research use measure 4 and 
measure 6. (See table 10b and Appendix H for full details of the results.) Importantly, having controlled 
for these teacher background characteristics, the association of the RLC intervention became non- 
significant for all 6 teacher research-use measures. This suggests that teacher characteristics such as 
leadership status and holding a postgraduate qualification are more important in promoting teachers 
research use than participating in the RLC intervention. Note that we also explored whether this result 
would change if interactions between intervention/control group and teacher characteristics were 
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additionally controlled for in the analysis, in other words, whether senior or better qualified teachers in 


the intervention schools may have responded more positively than those in control schools. However, 
this was not found to be the case. 
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Table 10b Multilevel modeling results for part 2 (teacher outcome analysis for measure 1: Positive disposition to academic research in informing 
teaching practice) 


(1) 7) (3) 
Standardized Standardized Standardized 
measure measure measure1 
score score score 
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Standard errors in parentheses 
“p<0.05,” p<0.01, p< 0.001 
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Table 11 shows the numbers of responses to the Teacher Research Use Survey in intervention and 
control schools in round two and round three. Tables 12 to 17 show the typical change in a school’s 
average teacher-reported use of research evidence to inform their practice between the end of the first 
year (round two) and the end of second year (round three) of the intervention. The unit of analysis is 
schools and in each case a two-way ANOVA was used (intervention/control and round two/round three). 
It should also be reiterated that the reliability for the teacher survey measures was relatively low except 
for measure 1 (positive disposition to academic research in informing teaching practice; see Table 2). 
It was found, for intervention schools but not for control schools, that there were noticeable differences 
between rounds two and three of the survey for five of the teacher measures. Teacher measures, on 
average, in intervention schools increased for positively worded measures (1, 2, and 5) and decreased 
for negatively worded measures (3 and 4) between the end of the first year (2015, round two) and the 
end of second year (2016, round three) of the intervention. In contrast, average teacher measures in 
control schools typically did not change between rounds two and three. These changes between the 
end of the first year and the end of the second year, however, were not statistically significant in the 
ANOVA analysis due to the relatively small number of schools. 


Table 11: Number of respondents and schools in the Teacher Survey by intervention/control 
and round two/round three 


Count: number of respondents/schools in teacher survey 


Round 2 Round 3 
Intervention 441/56 235/39 
Control 525/52 464/40 
Total 966/108 699/79 


Table 12: School means of teacher responses for Measure 1 (by intervention/control and 
round two/round three) and two-way ANOVA results4 for testing the differences between 
school means of Measure 1 


iN CEFeETUL ge Mm im ( oLeX-IL CMe Welt) Leki d(o) aM come: (er-(e(-Vunl em ccs-1-1- eM Maleolaiil tare m cere Coal iare mm el e-(ed(e-) B 


School means 


Round 2 Round 3 Total 
Intervention 0.04 0.15 0.08 
Control -0.04 -0.08 -0.05 
Total -0.001 -0.0002 
Note: 


e __ there is no statistically significant difference in school means of Measure 1 between round two and round 
three surveys (p = 0.67); 

e __ there is a statistically significant difference in school means of Measure 1 between intervention and control 
groups (p = 0.05); 

e the interaction term of the two factors (intervention/control; round two/round three) has no significant effect 
on school means of Measure 1 (p = 0.26); and 

e the three random randomisation variables were also controlled in ANOVA. 


4 In Tables 12-17, the calculation of means in each cell and two-way ANOVA were based on the average school 
scores for Measures 1-6. The school means were weighted by using the Analytic weights (file:///O:/using-weights- 
in-stata(1).pdf) in both calculation of means and two-way ANOVA analysis to adjust the bias caused by unequal 
respondents in different schools. 
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Table 13: School means of teacher responses for Measure 2 (by intervention/control and 
round two/round three) and two-way ANOVA results for testing the differences between school 
means of Measure 2 


WN CeXeRTU] ged CULT = Wo) wr Cor-(o (Van (em gek-y-r-1 ced gm COM Anco) dan me-y-1(=\e¢(e]ameyim (r-(eral ale pr-| °) el gey- (ea 1-1) B 
School means 


Round 2 Round 3 Total 
Intervention 0.04 0.15 0.08 
Control -0.04 -0.07 -0.05 
Total 0.001 0.002 
Note: 


e __ there is no statistically significant difference in school means of Measure 2 between round two and round 
three (p = 0.68); 

e there is no statistically significant difference in school means of Measure 2 between intervention and 
control group (p = 0.09); 

e the interaction term of the two factors (intervention/control; round two/round three) also has no significant 
effect on school means of Measure 2 (p = 0.28); and 

e the three random randomisation variables were also controlled in ANOVA. 


Table 14: School means of teacher responses for Measure 3 (by intervention/conitrol and 
round two/round three) and two-way ANOVA results for testing the differences between school 
means of Measure 3 


IM Ce¥eKTUT = wee Me ( of-)qer-) on d(o) alm dat-Lar-(er-(ol-yanl (em c-c-i-1- leet male) MOly-1 (01m Com (-r-leual late) 
Te} afoto) Man tst- Lats 


Round 2 Round 3 Total 
Intervention -0.03 -0.10 -0.05 
Control 0.02 0.05 0.04 
Total 0.005 -0.001 
Note: 


e __ there is no statistically significant difference in school means of Measure 3 between round two and round 
three survey (p = 0.70); 

e there is no statistically significant difference in school means of Measure 3 between intervention and 
control group (p = 0.07); 

e the interaction term of the two factors (intervention/control; round two/round three) also has no significant 
effect on school means of Measure 3 (p = 0.31); and 

e the three random randomisation variables were also controlled in ANOVA. 


Table 15: School means of teacher responses for Measure 4 (by intervention/control and 
round two/round three) and two-way ANOVA results for testing the differences between school 
means of Measure 4 


WN (exeKTUT gow ME ( ol-laer-) on d(ol am iar-l mel) damecveslere) me (ol-t-maleym-lalerelele-(e(-MOl--Mey m-Ler-(e(-1nil (em c-t-12t- een) 


School means 


Round 2 


Round 3 


Total 


Intervention 


-0.01 


-0.09 


-0.04 
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Control 0.01 0.04 0.03 
Total 0.001 0.0007 
Note: 


there is no statistically significant difference in school means of Measure 4 between round two and round 
three survey (p = 0.87); 

there is no statistically significant difference in school means of Measure 4 between intervention and 
control group (p = 0.45); 

the interaction term of the two factors (intervention/control; round two/round three) also has no significant 
effect on school means of Measure 4 (p = 0.41); and 

the three random randomisation variables were also controlled in ANOVA. 


Table 16: School means of teacher responses for Measure 5 (by intervention/control and 
round two/round three) and two-way ANOVA results for testing the differences between school 
means of Measure 5 


Measure 5 (active engagement with online evidence platforms): 
School means 


Round 2 Round 3 Total 
Intervention -0.03 0.07 0.005 
Control 0.03 -0.04 -0.003 
Total 0.004 -0.002 
Note: 


Education 


there is no statistically significant difference in school means of Measure 5 between round two and round 
three survey (p = 0.74); 

there is no statistically significant difference in school means of Measure 5 between intervention and 
control group (p = 0.64); 

the interaction term of the two factors (intervention/control; round two/round three) also has no significant 
effect on school means of Measure 5 (p = 0.11); and 

the three random randomisation variables were also controlled in ANOVA. 
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Table 17: School means of teacher responses for Measure 6 (by intervention/control and 
round two/round three) and two-way ANOVA results for testing the differences between school 
means of Measure 6 


WN CeFeKSUT CoM om (dae) a (sxele(- Mey mm a=t-1-r-laedamalavel lave fom-lave mri(citarelet) 


School means 


Round 2 Round 3 Total 
Intervention 0.09 0.02 0.07 
Control -0.08 -0.01 -0.05 
Total 0.001 -0.0002 
Note: 


e __ there is no statistically significant difference in school means of Measure 6 between round two and round 
three survey (p = 0.88); 

e there is no statistically significant difference in school means of Measure 6 between intervention and 
control group (p = 0.20); 

e the interaction term of the two factors (intervention/control; round two/round three) has no significant effect 
on school means of Measure 6 (p = 0.23); and 

e the three random randomisation variables were also controlled in ANOVA. 


Cost 


To buy in, an RLC over a year would cost £500 per day over six days, plus an average of £600 for travel 
and resources—a total of £3,600 for each RLC per year to be divided between the schools in the RLC. 
Social network analysis to determine who would be appropriate Evidence Champions was calculated 
as five days per RLC, at £500 per day, as a one-off cost to the RLC at the start of the intervention. 
Based on 14 RLCs across 59 schools, and a total of 19,158 children across all year groups in the 
schools, the cost per child is £3.24 per year for a three-year intervention (Table 18). 


Table 18: Cost of the intervention 


Cost 


Type of cost 


MWe) C1 exoy) mo)'(=) arc mm Me) (| evel) oL-lg 
years pupil per year 
over 3 years 


One-off social Start-up cost per | £2,500 per RLC £2,500 per RLC 

network analysis | RLC 

One year of RLC | Running cost per £3,600 per RLC £10,800 per RLC 
workshops RLC 

Total £13,300 per RLC | £3.24 


Each Evidence Champion (two per school) would be expected to attend four day-long workshops per 
year. The amount of time spent by Evidence Champions and other teachers engaging with research 
was determined by their school, and different schools approached this differently. In the majority of 
schools, teachers were expected to do this in addition to their normal timetables—few schools allocated 
extra time to Evidence Champions. Some schools allocated time in staff meetings to discuss the RLC 
project and activities, others did not. 


Schools in the evaluation received the intervention for free as the cost was borne by EEF. 
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Process evaluation 


This section reports on findings from case studies of six treatment schools and two control schools, 
information gathered from the Evidence Champion surveys at the end of Year 1 and the end of Year 2 
of the intervention, and information gathered from visits to the RLC workshops. Interviews in case study 
schools and Evidence Champion surveys enabled us to understand the ways in which the intervention 
was experienced by teachers, and how teachers and schools aim to use research evidence to inform 
practice. 


Discussions with the intervention team and attendance at workshops enabled the evaluation team to 
understand the content of the training. During the first year of the project this centred on issues such 
as: 


e looking at research on a given subject area; 

e understanding effective ways to share research; 

e understanding what baseline data is and how to collect it; 

e analysing data; 

e developing and trialling new approaches to teaching and learning that build on data; 
e looking at research on how to lead and manage change; 


e using Guskey’s (2000) model to understand how to think about, and measure, impact (this 
model proposes five levels of understanding professional development; participants’ reactions 
and participants’ learning; organisation support and change; participants’ use of new 
knowledge and skills; and student learning outcomes); 


e understanding theories of action and how these can be used to think about and implement 
change; 


e understanding how to assess impact; 
e effective ways of mobilising knowledge within and across schools; and 
e how to source high quality literature. 


Training during the second year of the project was more ‘light touch’ and centred on issues such as 
determining the focus of inquiry and desired outcomes, deciding on and analysing baseline data, 
determining potential actions, developing a roll-out plan, identifying measures of impact, analysing 
impact, and reflecting on ways forward. 


From the information gathered through the Evidence Champion survey, attendance at workshops was 
good, particularly in the first year. Most participants from this sample said they had been to either three 
or four sessions. Absences were mostly due to time constraints. According to interviewees, schools 
tried to ensure that at least one Evidence Champion or senior leader attended every workshop. 


In both years, workshops were very well received by participants: all of them deemed ‘very useful’, both 
by a majority of interviewees and by Evidence Champion survey respondents. In interviews, beyond 
the interest and usefulness of their topics, participants stressed they valued the workshops largely for 
the opportunity these afforded to meet teachers from other schools, share ideas, and reflect together 
about their practices. In this sense, training workshops effectively functioned as research learning 
communities, supporting the work done by the Evidence Champions in their respective schools, and 
promoting collaboration and reflection among colleagues. 


The choice of facilitators was crucial for the success of the workshops: in interviews, they were 
invariably well evaluated by participants, not just for the delivery of the sessions, but also for maintaining 
a line of communication (through phone and email) with the schools for the whole duration of the project: 
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‘A lot of emails and communication, [the facilitator] was lovely with support if we needed anything’ 
(teacher interview, School H, 2016). 


By giving advice, access to resources, and helping with the structuring of the interventions, facilitators 
were considered supportive and helpful, ensuring the continuity of the project in the schools that were 
taking part. 


Implementation 


Attendance by the evaluation team at a selection of RLC workshops suggested that they were run 
according to the planned agenda. Inevitably conversations took different turns within each workshop, 
and different participants responded differently to the content. This is an inherent part of running 
professional development sessions that aim to promote engagement with tools (in this case, research) 
and reflection on practice. 


Among the factors that were seen as relevant to the success of the project by interviewees, the most 
important was that schools already had a ‘research-friendly’ culture, as described by the headteacher 
in School A in 2015: 


‘That’s why it was useful using the staff meetings in that way, to say, “Well, let’s all come with the 
piece of research,” and you'd look very silly if you turned up with nothing and everyone else is 
having the discussions. And there were some people who’d made more effort than others, and 
that’s a good model then for the people who’d perhaps not made as much effort to see what the 
others had done.’ 


Prior to their participation in the RLC project, treatment case study schools showed varying levels of 
engagement with research. No school seemed to normally use published research to systematically 
inform teaching (beyond training events and INSET days). However, periodical revision of pupil data to 
monitor attainment, trialling new pedagogical approaches, and diverse forms of collaboration with other 
schools were repeatedly mentioned as common practices. These could be characteristic of a positive 
attitude towards the analysis and discussion of evidence, with collaboration with other schools being 
used as a means to share and discuss evidence. In this sense, according to interviewees, the RLC 
project confirmed, validated, reinforced, or enhanced the learning culture of these institutions. 


Interviewees suggested that other factors that helped the workings of the project were: 


e the existence of effective communication mechanisms among teachers and leadership team— 
regular staff meetings in particular; 


e the adoption of a ‘participatory’ approach, respecting the teachers' autonomy to manage their 
interventions; 


e aproactive staff, keen on trying initiatives to improve learning; 


e the size of the school (smaller institutions facilitated communication and the adoption of a 
whole-school approach to the project); and 


e continued support from the Institute of Education via workshop facilitators. 


Given the enthusiasm of schools to sign up to the project in the first place and the nature of the 
interviews, it appears that the intervention is very attractive to schools, especially those that already 
work together and so have existing collaborative systems. The intervention has the potential to effect 
considerable change for those who are Evidence Champions: workshop attendance and interviews 
over the two years of the project evidenced a shift in approach and increased confidence in using 
research to inform practice, and in discussing this with colleagues: 


‘Definitely, the peer and self-assessment is definitely having an impact. At the start of the year 
they would just have absolutely no clue, you know, “oh, well done, you've done really nice hand 
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writing, now you need to look at adding adjectives”, where they probably used hundreds of 
adjectives in their piece of work. And then, [...] each staff meeting, [the Evidence Champions] 
have done, they've kind of given us a new [ideas], so first it was to like model how to give peer 
assessment, and then it was to show them some good and bad examples of it and then talk about 
why those examples were good or were bad. And now | feel at the end of the year that the 
children really, you know, what they're looking for and how to give good feedback on their work 
here’ (teacher, School E, 2016). 


According to all interviewees, the biggest difficulty faced when implementing the project was the lack of 
time to engage with research given day-to-day teacher workloads, for example: 


‘Time... and it is very difficult to improve that. We are working our socks off as it is without doing 
anything additional, so obviously all of this is extra on top of what everyone, the Champions, the 
people on the ground, it is all additional. So you are relying very much on goodwill, of people 
giving up their time outside of what they are doing. Obviously as a teacher, you would hope that 
most people are reflective on ways to improve their practice but | think as a member of SLT, it is 
really important not to lose sight of the fact that you are already making a lot of demands on your 
teachers. And so it is about keeping that balance healthy, between what you are asking them to 
do in addition to what they are already doing’ (Evidence Champion, School G, 2015). 


This was particularly dramatic for Year 6 teachers (Some Evidence Champions among them), busy with 
the preparation of SATs and other end-of-KS2 activities, but the issue was raised by every teacher 
interviewed, regardless of his or her year group. This meant that the project worked better in schools 
where there were already instances (like regular staff meetings) and practices (collaboration, revision 
of pupil data, class interventions) whereby project activities could be channelled, without adding too 
many extra demands on teachers. Most teachers needed to perceive an added value or a concrete 
impact of this project before considering finding time for it within their regular working schedule. 


Moreover, a few schools went through Ofsted inspections during the implementation of the project, 
having to prioritise accountability activities over pedagogical development. Similarly, some interviewees 
claimed that the government's changing educational policies pressured them to focus on short-term 
initiatives, distracting them from developing a professional culture in their schools. 


High staff turnover was another problem that affected some schools, impairing the continuity and 
stability of the project. This was evident in case study schools, and through visits to the workshops 
where non-case study schools were also in attendance. In some cases this was because of maternity 
or sick leave, or increased in-school responsibilities, as highlighted by a teacher in School C in 2016: 


‘It's just the time restraints and obviously being out of class for a whole day is not ideal when 
you're in teaching and when you're in class, especially in Year 6, which is why I| had to drop out 
of it this year because !| couldn't afford to spend time away from the children.’ 


Other teachers moved to other jobs. This is a common situation in schools that calls for a sensible 
choice of Evidence Champions and early engagement of the whole staff. 


Varying levels of prior engagement with research meant some teachers in case study schools had 
inadequate skills or experience in order to deal with their interventions, needing extra support from 
Evidence Champions or colleagues, and slowing down the implementation of the project. Both Evidence 
Champion surveys and case studies revealed that the lack of open access to relevant, up-to-date 
research in most schools is a barrier to the development of the project, as many participants had to rely 
informally on colleagues doing university courses, or used non-academic sources to obtain information 
to feed their interventions. It was suggested that school subscriptions to research databases could 
boost the effectiveness of a project of these characteristics, although time and training in how to use 
these resources may be needed, particularly for those school staff who have not recently attended 
university. 
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One question explored by the process evaluation was whether the duration of the intervention was 
appropriate. It was apparent that two years are needed for teachers to develop their understanding and 
practice of engaging with, and using, research, for example: 


‘| think they're [the RLCs] better for two years. | think you always want to... because you want to 
have that research element. So it’s not only reading something, it’s how you then apply that 
reading to your situation. And | think you need to be able to explain it convincingly to the rest of 
the staff, that something is worth having a go at. And so, | think it does take the two years to have 
the first trial on a small scale, and then the second year, implement it out. And then, only after 
that second year can you really see if anything’s happening’ (Evidence Champion, School E, 
2016). 


‘We were saying, and agreeing, that to do a project and research the impact, you need to have 
more time, yeah. In a year, you’re almost introducing things or applying things. In the second 
year, you’re almost consolidating some of the things and seeing the impact. And | think that’s 
why | want to continue on for the following year, so that’s two years for me in my year group. 
Although it’s been two years within the school, but that’s in different year groups. | think two years 
in year 2, would be much for me, something | just want to continue on, yeah’ (Evidence 
Champion, School C, 2016). 


Further, the interviews during the second year of the research demonstrated a more reflective approach 
to research than those in the first year. 


Fidelity 


Interview and observation evidence from case-study visits indicates that the intervention was delivered 
as intended, although not all schools were able to attend all workshops due to time constraints, as 
mentioned above. Furthermore, staff turnover (evidenced in surveys, interviews, and attendance by the 
evaluation team at workshops) means that the Evidence Champion role may be played by different 
people over the course of two years, which may hinder the development of research expertise in a 
school. Data on workshop attendance for 45 schools (including one that dropped out before the start of 
the intervention and one that dropped out part way through) across ten RLCs were provided by the 
intervention team (see Table 19), with incomplete or no data being provided for four RLCs. 


Table 19: Number of schools attending different numbers of workshops (note that one school 
dropped out before workshops started) 


Number of workshops attended 5 6 7 8 


Number of schools attending this 2 2 6 34 
many workshops 


Of the records available across all RLCs, 300 workshop attendances involved two staff members, and 
79 workshop attendances involved one staff member. This indicates that the majority of schools sent 
two staff members to a workshop the majority of times, but this was by no means universal. 


Outcomes 


Many case study participants claimed that they perceived an impact in the classroom (that is, in 
children's learning) where interventions were implemented, for example: 


‘The children are a lot more... they know when they've done something well and they can tell you 
when they've done something well and they’re a lot more receptive to ways to improve now, and 
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can, and Key Stage 2 or Key Stage 1 are able to identify what they need to be doing next to make 
it even better. And | think it’s developed, like the school is developing a school ethos of like 
continuously growing and thinking about what they can do to make it even better’ (teacher, School 
E, 2016). 


The effect of the RLC project is, however, mostly evident in the positive attitude and improved 
engagement with research that participants showed in interviews and Evidence Champion surveys. 
Several participants declared a greater interest in using published research to support classroom 
interventions or broaden their understanding of their practice. Among different research-related 
practices, reading papers from academic journals, and discussing them with colleagues, had become 
more common in some schools: 


‘If you read a journal article and then, two weeks later, do something related to it, it’s not going 
to have the impact that you want. So it’s important to read that article, digest it, think about what 
you want to take from it all the time, rather than just read it and put it to one side. And that’s what 
we've been doing with this project, making sure we're reading and we’re taking bits from it so 
that we can use it because otherwise, if you’re not, it’s just a waste of time, really. You need to 
make sure that you are benefiting from that research and that, ultimately, the children will benefit 
as well’ (teacher, School A, 2015). 


Participants also suggested they now approach these texts more critically, considering the reliability of 
the sources, methodologies, and relevance for their own teaching. 


In a similar vein, other participants claimed the project had made them more reflective about teaching 
and more aware of their professional development. Engaging with research was seen as a part of 
striving for improvement in teaching and learning. Some suggested they are keener now on collecting 
evidence of the effects of their initiatives and adopting more formal procedures for trialling new 
approaches in the classroom, time permitting. 


‘It’s nice that colleagues do work amongst themselves so | know that with the Research Learning 
Communities you have four [members of staff] who have worked really closely together at looking 
and analysing about analysing about the Talk for Writing and scaffolding of learning, especially 
with that particular cohort, | think it’s something that’s probably really benefitted them in terms of 
that great depth of thinking and in terms of their delivery of the English curriculum—which has 
obviously been informed by research because that kind of forms part of the Research Learning 
Communities, so the fact that a teacher who isn’t involved in that project has managed to get the 
benefits of working with a colleague that has, | think has probably had a great impact on that 
curriculum up there’ (headteacher, School G, 2016). 


This quote also highlights the importance of sharing ideas with colleagues and developing a culture of 
collaboration. In interviews, this was often mentioned as an effect of the project: having established or 
reinforced instances of discussion among teachers, both within and between schools (in institutions 
that are members of school federations, for example). Given time constraints around implementing 
classroom interventions on a wider scale, cementing at least a culture of discussion and reflection with 
colleagues, oriented to pedagogical improvement and professional growth, is one of the most valued 
outcomes of the project. 


All these attitudes imply a positive approach towards, and understanding of, how research can inform 
educational practice. Whatever their level of prior acquaintance with research practices, it can be 
inferred from interviews that the project strengthened—among many participants—the idea that 
research has to be based on evidence (either published or self-collected), that it has a purpose 
(investigating a hypothesis, proving a conjecture, changing a pedagogical approach, broadening 
understanding), and that it involves reflection and collaboration with colleagues. The interviews 
demonstrated that research is viewed now by many teachers as a support for changing and improving 
educational practices, and ultimately benefitting children's learning. 
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Unintended or negative outcomes were not evident from interviews. As mentioned above, problems 
were faced due to staff turnover, and in some cases the wide geographical spread of schools meaning 
that Evidence Champions found it difficult to feel part of a cohesive RLC with little incentive to develop 
long-term relationships. Outcomes mentioned by schools, however, were about the extent to which 
schools were developing their evidence-informed practice, and the amount of collaboration with other 
schools. 


Formative findings 


The delivery of the intervention itself was generally seen as positive. The majority of suggestions for 
improvement related to school systems. When asked about improvements, almost all case study and 
Evidence Champion survey participants replied that the intervention could be improved by schools 
allocating more time to Evidence Champions to support their work on the project, including reading, 
discussion, and feedback with colleagues in the school. For example: 


‘Allocating a time each week or fortnight to actively think about what we are doing. It has tended 
to end up being at the end of a long list of essential “to dos” rather than as a priority’ (Evidence 
Champion Survey, 2015). 


Several participants felt that access to (and the time to read) more research would be helpful, for 
example through online journals. Some participants also mentioned that they would welcome the 
opportunity to read more research in-depth, although others felt that summaries would be preferable. A 
few participants mentioned that it would be good for all schools within an RLC to be focusing on the 
same project to enable the support to be more tailored, for example: 


‘Linking with other schools would have been more beneficial if a similar project/theme had been 
undertaken, as it would have been more collaborative rather than separate’ (Evidence Champion 
survey, 2016). 


However, this needs to be balanced with the requirement for participants to ‘buy in’ to the project: 
schools and teachers are more likely to commit to a project that fits with their own school agenda, and 
schools in the same RLC may not all have the same needs. Matching schools by project focus may be 
difficult to reconcile with the requirement for geographical proximity: several participants mentioned this 
in the context of building more cohesive networks and facilitating travel to workshops. 


Control group activity 


One of the two case-study control schools had a strongly research-oriented culture in place. The 
headteacher in this school promoted engagement with research, reflection on how it could inform 
practice (research articles were distributed by the headteacher prior to staff meetings for discussion), 
and supported teachers to use their own classroom data to develop their practice. Two of the senior 
leadership team were also part of networks that engaged with research to inform and develop practice. 
Although less structured than an RLC, it seems that this school was working with a similar ethos to that 
promoted by the intervention. This arose partly from their local network of schools, and partly from the 
headteacher’s belief in the importance of research-informed practice. However, access to a wide range 
of good-quality research was problematic, and some teachers expressed a lack of confidence in their 
ability to evaluate published research. 


The other case-study control school was less actively engaged with research: they did not have 
mechanisms for engaging with or sharing published research, and it tended to take a low priority 
because of other time commitments. Staff felt that research had the potential to inform their practice 
but relied on senior leaders to tell them how they should implement ideas. This school was only visited 
once, in the first year of the evaluation, and at this point the headteacher expressed dissatisfaction that 
the school was in the control group because they had wanted the opportunity to link with other schools. 
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The two control schools were chosen because of their different levels of engagement with research in 
the first place. It is interesting to reflect on whether they would have responded differently to the 
intervention: in the first of the control schools discussed, the teachers were in a position to immediately 
engage with the intervention, with supporting mechanisms within the school. Whether the second 
control school would have found the capacity to engage fully with the intervention is questionable— 
although being in the intervention may have encouraged them to place more of a priority on research. 
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Conclusion 


Key conclusions 


1. The project found no evidence that Research Learning Communities improves reading outcomes 
for children at Key Stage 2. 


The project did find a positive impact on teachers’ disposition towards research. There was, 
however, some evidence that this impact may have been influenced by other factors such as the 
level of postgraduate qualifications or seniority of teachers that took part in the intervention. 


Exploratory analysis identified some evidence of a small positive relationship between teachers’ 
disposition towards research and pupil outcomes, irrespective of involvement in an RLC. 


Evidence Champion roles in each school were intended to be held by the same people throughout 
the intervention, to support the development of a research-focused culture within each school. 
Staff turnover was therefore a barrier to implementation. 


Some teachers felt that it may take a number of years for participation in an RLC to change 
teaching practice and improve pupil outcomes. Future research could therefore examine longer 
term impacts. 


Interpretation 


The project has demonstrated that it is possible to overcome the barriers to research use cited by 
numerous researchers (for example, Speight et a/., 2016; Nutley et a/., 2007; Sharples, 2013; Hemsley- 
Brown and Sharp, 2003), including teachers’ ability to interpret research evidence, through a systematic 
intervention that seeks to transform evidence in a similar way to that suggested in the pilot study of 
Nelson et al. (2017). However, while the duration of the two-year intervention was suggested by 
teachers to be appropriate, the duration of this evaluation is problematic in that the processes involved 
in teacher learning, its impact on teacher practice, and from there the impact on pupil outcomes (as 
highlighted in the different stages of Guskey’s (2000) model), are likely to take place over the longer 
term. 


The intervention demonstrated that it is possible to empower teachers to develop a more confident and 
reflective approach to using research to inform their practice (as advocated by Sharples, 2013), and to 
enable evidence-informed practice to emerge in the broadest sense, using existing research to 
stimulate thinking, and collecting evidence within schools to build on that learning. Although we need 
to acknowledge that this evidence largely stems from teacher self-reports rather than an independent 
observation of behaviour change. The analysis of KS2 outcomes demonstrates statistically significant 
effect sizes of 0.07 (GPS) and 0.09/0.08 (GPS/numeracy) respectively for Measures 1 and 2, for one 
standard deviation unit of teachers’ reported use of research evidence. This indicates a small positive 
association between teachers’ reported use of research evidence, at least for the secondary pupil 
outcomes examined, irrespective of whether they are in an intervention or control school. The causal 
nature of this association is, of course, unclear; it may just be that teachers who teach well are also 
likely to engage with research more, and that there is no direct causal link between the two. Combined, 
however, with the tentative evidence that the intervention had a positive impact on teachers’ reported 
use of research evidence to inform their practice at the end of the two year RLC period (effect size 
0.20-—0.21 for Measures 1 and 2), this indicates that in the long term this intervention may have potential 
to support pupil outcomes of subsequent cohorts of pupils through teacher development. The RLC 
dosage (how many workshops were attended by Evidence Champions in a school) also approached 
significance in terms of pupil outcomes for KS2 GPS, which lends further tentative support to this claim. 
In this case, the effect size was 0.02 for each additional workshop attended. 
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Limitations 


The goal of the intervention was to encourage teachers to develop research-informed practice. 
Measuring pupil outcomes to assess the impact of the intervention presents some problems. The 
(simplified) steps of the intervention could be seen as: changing teachers’ attitudes towards using 
research to inform practice; developing teachers’ understanding of how to use research to inform 
practice; and supporting teachers to use research to inform practice. From there, it is expected that a 
change in teachers’ practice will, over time, impact on pupil learning and then on pupil outcomes. To 
expect this whole process to happen over the course of two years is ambitious and therefore we are 
not surprised that there were no significant main effects of the intervention on pupil outcomes. Indeed, 
in this context it is very encouraging that there is even a small effect of teachers’ research use on KS2 
outcomes. 


Further, it is important to consider the substantive focus of research that teachers engage with, and the 
focus of their development of practice. For teachers to be motivated to engage with research, they need 
to have some buy-in to the substantive topic, so it is important to allow them some agency in determining 
their own focus. To make the process of sourcing existing research evidence manageable, the delivery 
team presented RLCs with a range of substantive topic options, including, but not limited to, those with 
a literacy or numeracy focus. The eventual foci chosen by RLCs included those around feedback, 
marking and moderation, growth mindsets, literacy, numeracy, and setting (or some combination of 
these). However, the nature of pupil outcome measurement means that we had to use pupil outcomes 
in particular curriculum areas—areas that may not have matched the substantive focus of an RLC. To 
address this, analysis included a variable of the RLC’s focus—which also yielded no significant effects. 
The RLC focus is, of course, also problematic as a measure: not all schools were focusing specifically 
on literacy or numeracy, and in addition, some may have been focusing on a more generic topic (for 
example growth mindsets) but applying this to (for example) teaching practice in numeracy. The pupil 
outcomes that an individual school is focusing on may differ to the way the project is framed in the RLC 
as a whole—and these outcomes may not directly relate to reading, numeracy, grammar, punctuation, 
and spelling. In addition, some schools changed their substantive focus from one year to the next (as 
was evident in case study schools and RLC observations). 


RLCs aim to support teachers to develop and reflect critically on their own practice through engagement 
with evidence. These skills are developed through teachers learning to read and understand existing 
research evidence, and using that learning to evaluate needs in their own school, develop strategies to 
meet those needs, and evaluate the implementation of those strategies—in short, develop evidence- 
informed action research cycles. The approach is not just about teachers taking interventions that have 
been found to be effective in other contexts and applying them in their own school; it is about developing 
amore critical perspective on practice. In an RLC, teachers focus on research around issues that they 
believe need addressing in their school, and through that engagement with evidence, develop their own 
ideas for how to develop their practice. The developers selected evidence that they deemed to be of 
high quality on each topic, and summarized it in such a way that it was accessible for teachers. This 
was then used as a basis for discussion within the workshops, to support the development of teachers’ 
critical engagement with research. 


The extent of missing data in the analyses involving teacher outcomes should also be considered a 
limitation. While the balance of characteristics of treatment and control schools was not substantially 
affected, this resulted in a reduction of sample size and subsequently in power for Part 2 analysis linking 
pupil outcomes to teacher outcomes, and the analysis of teacher survey outcomes. Nonetheless, the 
scope of the trial and the diversity within the sample in terms of characteristics suggests that the findings 
will be relevant to a wide range of schools. 


Future research and publications 
Future research is needed to explore the relationship between teacher background characteristics and 
their response to the intervention. In addition, it is important to explore the different possibilities afforded 


by the role of Evidence Champions in terms of developing a whole-school research culture, and how 


Education Endowment Foundation 46 


Research Learning Communities 


this can work in different types of school with different existing orientations to research. The case studies 
highlighted that the role of Evidence Champion worked in different ways for different staff members, 
and for different schools, but did not allow for a systematic exploration of this. Also among the factors 
that were seen as relevant to the success of the project by interviewees, the most important was that 
schools already had a ‘research-friendly’ culture in the schools, so further research on this aspect would 
be helpful. Most crucially, however, more research is needed to understand the longer-term impact of 
engagement with research and a developing whole-school research culture on teachers’ practice and 
the outcomes of subsequent cohorts of pupils taught by participating teachers. 
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Appendix A: EEF cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. More information about the EEF’s approach to cost evaluation can be found here. Cost 
ratings are awarded as follows: 


Cost rating | Description 


Very low: less than £80 per pupil per year. 
Low: up to about £200 per pupil per year. 


Moderate: up to about £700 per pupil per year. 


High: up to £1,200 per pupil per year. 
Very high: over £1,200 per pupil per year. 
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Appendix B: Security classification of trial findings 


4a Fair and clear quasi- 
experimental design for 
comparison (e.g. RDD) with MDES < 7 
appropriate analysis, or 0.3 11-20% Adjustment 
experimental design with for Balance 
minor concerns about validity [0] 
eee Oe 
Well-matched comparison 
(using propensity score 
matching, or similar) or MDES < 
experimental design with 0.4 21-30% 
moderate concerns about 
validity 
Adjustment 
2a for threats [~~ 
Weakly matched comparison to internal 
or experimental design with MDES < 31-40% validity 
major flaws 0.5 
[0] 
18 ah ee (| (| 
Comparison group with poor 
or no matching (E.g. MDES < 41-50% 
volunteer versus others) 0.6 
a 
No comparator ae BY one ROGGE 


Initial padlock score: lowest of the three ratings for design, power and attrition = 5 padlocks 
Reason for adjustment for balance (if made): N/A 

Reason for adjustment for threats to validity (if made): N/A 

Final padlock score: initial score adjusted for balance and internal validity = 5 padlocks 


The design is a randomised controlled trial. MDES is 0.18 at randomisation, pupil level attrition is 
approx. 9%. The data is balanced on pre-test pupil attainment and there are no substantial threats to 


validity. 


5 Attrition should be measured at the pupil level (even for clustered trials) and from the point of randomisation to 
the point of analysis. 
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Appendix C: MoU 


Agreement to participate in the 
Research Learning Communities project and evaluation 


Please sign both copies, retaining one and returning the second copy to Dr Chris Brown. You can do 
this either by post (please send the signed copy to: London Centre for Leadership in Learning, Institute 
of Education, 20 Bedford Way, London WC1H OAL) or by email (chris.brown@ioe.ac.uk). 


School Name: 


Aims of the Evaluation 


The aim of this project is to evaluate the impact of Research Learning Communities - an approach 
designed to improve schools’ awareness and use of research. The evaluation will look at both changes 
in teaching practice and Key Stage 2 outcomes. The results of the project will contribute to our 
understanding of what works in improving teaching and learning and will be widely disseminated to 
schools in England. Ultimately we hope that the evaluation will equip school staff with the skills needed 
to be able to use research to enhance their practice and so improve learning outcomes for all pupils. 


The Project 


Starting in September 2014, the London Centre for Leadership in Learning (LCLL) at the IOE will be 
collaborating with some 100+ primary/junior schools in a two year project to explore ways of using 
evidence to support improvement at Key Stage 2. As part of the project, we will: 
e Identify and recruit 2-3 ‘evidence champions’ from each school, at least one of whom will be 
the head teacher or another senior leader. 
e Work with the evidence champions in Research Learning Communities (RLCs). Each RLC will 
bring together evidence champions from around five schools. Evidence champions will attend 
eight Research Learning Community workshops between September 2014 and July 2016 (i.e. 
four workshops each school year). Workshops will be designed to help evidence champions 
examine research and evidence relating to an agreed area of focus, and to develop, apply and 
evaluate school key stage-wide improvement strategies based on this evidence. 


e Ensure between workshops that evidence champions are provided with intersessional tasks to 
guide their evidence sharing activity and to help them effectively implement their improvement 
strategies. Extensive support will also be provided to help schools with these tasks. 


e Build capacity to ensure the sustainability of the approach, meaning that schools can continue 
to run RLCs and use evidence effectively, after the project ends. 


Structure of the Evaluation 


The evaluation is being conducted by the University of Bristol. Schools who agree to take part are 
randomly allocated to either the intervention group or a wait-list control group. This allocation will 
happen in September 2014. Random allocation is essential to the evaluation as it is the best way of 
outlining what effect Research Learning Communities have on children’s attainment. It is important that 
schools understand and consent to this process. 


Following the allocation of schools into intervention or control: 
e The schools in the intervention group will receive support from LCLL, through RLCs in the 
2014/205, and 2015/2016 academic years. 


e The schools in the control group will receive free access to RLCs from the 2016/17 academic 
year. In addition, they will receive a financial incentive of £1,000 at the end of the 2015/16 
academic year, for their full participation in the evaluation. 
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Pupil attainment at KS2 literacy and numeracy in 2015/16 will be used for the evaluation. The attainment 
of all pupils in this cohort in all the schools in the evaluation will be used. In addition, 85% of all teachers 
(including all KS2 teachers) of all RLC and control schools in the evaluation will complete three surveys 
about their awareness, understanding and use of research. These surveys will be collected in July 
2014, June 2015, and June 2016. Interviews will be conducted with school staff at two time points 
across the project in eight schools (six RLC schools and two control schools). 


Use of Data 


Pupils’ KS2 scores and any other pupil data will be treated with the strictest confidence. Pupils’ KS2 
scores and other data will accessed through the National Pupil Database by the University of Bristol. 
This will be shared with the London Centre for Leadership and Learning, and with EEF. No individual 
school or pupil will be identified in any report arising from the research. 


Responsibilities 


In running this project, the role of the London Centre for Leadership in Learning will be to: 
Identify the key area of focus for each RLC each year and identify research and evidence 
relating to this focus. 


2. Deliver eight workshops for each Research Learning Community over the course of the 
project (September 2014 — July 2016: i.e. four workshops each year) and within these work 
with RLCs to examine relevant research and evidence and to help RLCs develop, apply 
and evaluate school or key stage-wide improvement strategies based on the evidence. 

3. Develop intersessional activity that helps evidence champions share evidence within their 
schools and to apply and evaluate school or key stage-wide improvement strategies 


4. Provide on-going support to the evidence champions to help with intersessional activity 
5. Be the first point of contact for any questions about the evaluation 


The University of Bristol will: 
i. Conduct the random allocation. 


ii. | Collect and analyse all the evaluation data from the project 
iii. | Disseminate research findings 


The role of participating schools will be to: 


1. Identify a member of the leadership team to take overall responsibility for the success of the 
initiative in school and to ensure the completion of all required reporting. 


2. Consent to random allocation and commit to the outcome (whether RLC or control). 


3. RLC schools will commit to providing 2-3 ‘evidence champions’ and ensuring these are 
released to the project. At least one evidence champion will be the headteacher or another 
senior leader and one an identified opinion leader (see below): i.e. a teacher who is frequently 
turned to for advice and support with regards to teaching and learning. The evidence champions 
will need to be attend all four workshops each year and will also need to be released to and 
supported in developing their agreed improvement project and in engaging intersessional 
activity. 

4. RLC schools will ensure all their teachers complete evaluation surveys in June 2014, June 
2015 and June 2016. RLC schools will also need to ensure their teachers complete a ‘Social 
Network Analysis’ survey in September 2014 to enable the research team to ascertain key 
opinion leaders within schools. 


5. RLC schools will take it in turns to host RLC workshop sessions. 


6. Control schools will ensure their teachers complete evaluation surveys in June 2014, June 2015 
and June 2016. 

7. RLC and control schools will provide a 2015/16 class list of Year 6 pupils using UPNs and 
linked to the teacher. 


8. Headteachers from both RLC and control schools need to indicate their preferred areas of 
improvement in relation to improving KS2 outcomes so that LCLL can ensure each RLC has a 
common focus area. 


Education Endowment Foundation 53 


Research Learning Communities 


9. Headteachers should inform Dr Chris Brown if they are involved in any other EEF funded 
projects. 


Please note that schools will need to provide the following data / information in order to be considered 
for the project: 


e This agreement signed by the school Head and Chair of Governors 

e The evaluation survey issued in June 2014 to be completed by 85% of teachers (including all 
KS2 teachers) see point 4, above. 

e A 2014/15 class list of Year 5 pupils using UPNs and linked to the teacher. 


We commit to the evaluation of Research Learning Communities as detailed above 


Head teacher [NAME]: 


Headteacher email: 


Chair of Governors [NAMES]: 


Date: 
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Appendix D: Letters of consent 


D.1 Teacher information letter prior to survey 


Research Learning Communities Project Evaluation 


Dear Teacher 


You may be aware that your school is participating in the Research Learning Communities project, run 
by the Institute of Education (IOE). Researchers from the Graduate School of Education at the 
University of Bristol will be conducting an evaluation of Research Learning Communities. As a teacher 
at a school participating in the Research Learning Communities project, we would like to invite you to 
participate in the evaluation. 


Aims of the Evaluation 


The aim is to evaluate the impact of Research Learning Communities - an approach designed to 
improve schools’ awareness and use of research. The evaluation will look at both changes in reported 
teaching practice and at Key Stage 2 outcomes. The results of the project will contribute to our 
understanding of what works in improving teaching and learning and will be widely disseminated to 
schools in England. Ultimately we hope that the evaluation will equip school staff with the skills needed 
to be able to use research to enhance their practice and so improve learning outcomes for all pupils. 


The Project 


Starting in September 2014, the London Centre for Leadership in Learning (LCLL) at the IOE will be 
collaborating with some 100+ primary/junior schools in a two year project to explore ways of using 
evidence to support improvement at Key Stage 2. As part of the project, LCLL will: 


e Identify and recruit 2-3 ‘evidence champions’ from each school, at least one of whom will be 
the head teacher or another senior leader. 


e Work with the evidence champions in Research Learning Communities (RLCs). Each RLC will 
bring together evidence champions from around five schools. Evidence champions will attend 
eight Research Learning Community workshops between September 2014 and July 2016 (i.e. 
four workshops each school year). Workshops will be designed to help evidence champions 
examine research and evidence relating to an agreed area of focus, and to develop, apply and 
evaluate school key stage-wide improvement strategies based on this evidence. 


e Ensure between workshops that evidence champions are provided with intersessional tasks to 
guide their evidence sharing activity and to help them effectively implement their improvement 
strategies. Extensive support will also be provided to help schools with these tasks. 
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e Build capacity to ensure the sustainability of the approach, meaning that schools can continue 
to run RLCs and use evidence effectively, after the project ends. 


Structure of the Evaluation 


The evaluation is being conducted by the University of Bristol. Schools who agree to take part in the 
project will be randomly allocated to either the intervention group or a wait-list control group. This 
allocation will happen in September 2014. 


The schools in the intervention group will receive support from LCLL, through RLCs in the 2014/2015, 
and 2015/2016 academic years. The schools in the control group will receive free access to RLCs from 
the 2016/17 academic year. In addition, those schools in the control group who participate fully in the 
evaluation will receive a financial incentive of £1000 at the end of the 2015/16 academic year. 


The evaluation data collection will comprise three main strands: 


e Pupil attainment at KS2 literacy and numeracy in 2015/16. These will be accessed using Unique 
Pupil Numbers provided by schools. The attainment of all pupils in this cohort in all the schools 
in the evaluation will be used. 


e All teachers in all RLC and control schools in the evaluation will be asked to complete three 
short surveys about their awareness, understanding and use of research. These surveys will 
be collected in July 2014, June 2015, and June 2016. The survey to be collected in July 2014 
will take around 5 minutes to complete; the following surveys will take around 15-20 minutes to 
complete. 


e To understand how schools engage with RLCs, interviews will be conducted with selected 
school staff at two time points across the project in eight schools (six RLC schools and two 
control schools). Interviews will take between 30 and 60 minutes each. 


Use of Data 


Pupils’ KS2 scores and any other pupil data will be treated with the strictest confidence. Pupils’ KS2 
scores and other data will accessed through the National Pupil Database by the University of Bristol. 
This will be shared with LCLL, and with EEF. No individual school or pupil will be identified in any report 
arising from the research. Fully anonymised data from the teacher surveys will be shared with the 
National Foundation for Educational Research, who have developed the survey instruments. 


Right to withdraw 


Although it is important for the evaluation to work with as full a dataset as possible, participation in the 
evaluation data collection is voluntary for teachers. You have the option not to take part in the study, 
and we will not share the names of those who decline to take part with their schools. You can choose 
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to withdraw at any point in the study, for example when asked to take part in a later survey, or if asked 
to take part in an interview. If you complete a survey and then change your mind about participation, 
you do not have to submit your responses. If you complete an interview and then change your mind 
about participation, please let us know by July 2016 (when we will be completing the data collection). 


If you decide not take part in the evaluation, or any part thereof, please contact Jo Rose at the University 
of Bristol, with your name, class name/number, and school, using the contact details below. Any 
teachers that choose not to take part will not be identified to their schools. In the meantime, if you have 
any questions or queries please contact the evaluation team at the University of Bristol using the contact 
details below. 


We look forward to working with you over the coming two years. 


Best wishes 

Dr Jo Rose jo.rose@bristol.ac.uk 
Professor Sally Thomas s.thomas@bristol.ac.uk 
Dr Tim Jay tim.jay@bristol.ac.uk 
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D.2 Teacher baseline survey consent 


EEF RLC Evaluation Teacher Baseline Survey: Text to be used as intro/consent 


Welcome 


Thank you very much for agreeing to take part in this survey. This survey is part of a project 
commissioned by the Education Endowment Foundation which is exploring how schools improve their 
teaching and learning. The survey was developed by the National Foundation for Educational 
Research, and responses are being collected by a team from the Graduate School of Education, at the 
University of Bristol. We'd like to start by asking a few general questions on the types of information 
you use to inform decisions around teaching and learning in your school. There are no right and wrong 
answers - we are interested in hearing about your views and experiences in this area. 


The survey is short, comprising three main questions and some background information, and should 
take no more than ten minutes to complete. Your answers will be treated confidentially so you and your 
school will not be identified in any reports produced from this research. 


We will ask you to provide your name and class taught as part of this survey. This is so that we can link 
your responses to pupil data which is also being collected as part of the project. However, once you 
have submitted your responses you will then be assigned a code which will be used instead of your 
name: There will be a separate file linking your name to the code. This is to help ensure that all 
responses remain anonymous. 


Participation in this survey is voluntary. By completing and submitting this survey you are implying that 
you have given consent for your data to be used anonymously as part of the project. There will be a 
button to click at the end of the survey, when you can choose to submit your data. Please submit your 
responses by the end of July 2014. 
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D.3 Teacher survey consent 


EEF RLC Teacher Main Survey Consent: Text used as front page of online survey 


Welcome 


Thank you very much for agreeing to take part in this survey. 


In this survey we are going to ask you some questions about research in general, your participation in 

and the RLC project in 
general. There are no right or wrong answers - we are interested in hearing about your views and 
experiences in these areas. 


Your answers will be treated confidentially so you and your school will not be identified in any reports 
produced from this research. 


We will ask you to provide your name as part of this survey. This is so that we can link your responses 
to pupil data which is also being collected as part of the project. However, once you have submitted 
your responses you will then be assigned a code which will be used instead of your name. There will 
be a separate file linking your name to the code. This is to help ensure that all responses remain 
anonymous. 


Participation in this survey is voluntary. By completing and submitting this survey you are implying that 
you have given consent for your data to be used anonymously as part of the project. There will be a 
button to click at the end of the survey, when you can choose to submit your data. 


The survey is likely to take around 30 minutes to complete. 


Please submit your responses by (DATE). 
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D.4 Evidence Champion survey consent 


Evidence Champion Survey Consent: Text used as front page of online survey 


Welcome 


Thank you very much for agreeing to take part in this survey. 


In this survey we are going to ask you some questions about research in general, your participation in 
the Research Learning Community (RLC) workshops, your role as Evidence Champions and 
the RLC project in general. There are no right or wrong answers - we are interested in hearing about 
your views and experiences in these areas. Your answers will be treated confidentially so you and your 
school will not be identified in any reports produced from this research. 


We will ask you to provide your name as part of this survey. This is so that we can link your responses 
to pupil data which is also being collected as part of the project. However, once you have submitted 
your responses you will then be assigned a code which will be used instead of your name. There will 
be a separate file linking your name to the code. This is to help ensure that all responses remain 
anonymous. 


Participation in this survey is voluntary. By completing and submitting this survey you are implying that 
you have given consent for your data to be used anonymously as part of the project. There will be a 
button to click at the end of the survey, when you can choose to submit your data. 


The survey is likely to take around 30 minutes to complete. Please submit your responses by (DATE). 
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Appendix E: Teacher information and consent for interviews 


Research Learning Communities Project Evaluation 


Information sheet for Staff from schools or colleges participating in the Research Learning 
Communities Project Evaluation Interviews 


Dear Teacher 


You may be aware that your school is participating in the Research Learning Communities project, run 
by the Institute of Education (IOE). Researchers from the Graduate School of Education at the 
University of Bristol are conducting an evaluation of Research Learning Communities. As a teacher at 
a school participating in the Research Learning Communities project, we would like to invite you to 
participate in an interview as part of the evaluation. During the interview, we would like to discuss ways 


in which you engage with research as part of your role in school, and your thoughts on the Research 
Learning Communities project (NB This last part only if interviewing staff in RLC school). 


Aims of the Evaluation 


The aim is to evaluate the impact of Research Learning Communities - an approach designed to 
improve schools’ awareness and use of research. The evaluation will look at both changes in reported 
teaching practice and at Key Stage 2 outcomes. The results of the project will contribute to our 
understanding of what works in improving teaching and learning and will be widely disseminated to 
schools in England. Ultimately we hope that the evaluation will equip school staff with the skills needed 
to be able to use research to enhance their practice and so improve learning outcomes for all pupils. 


The Project 


The London Centre for Leadership in Learning (LCLL) at the IOE are collaborating with some 100+ 
primary/junior schools in a two-year project to explore ways of using evidence to support improvement 
at Key Stage 2. As part of the project, LCLL will: 


Identify and recruit 2-3 ‘evidence champions’ from each school, at least one of whom will be the head 
teacher or another senior leader. 


Work with the evidence champions in Research Learning Communities (RLCs). Each RLC will bring 
together evidence champions from around five schools. Evidence champions will attend eight Research 
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Learning Community workshops between September 2014 and July 2016 (i.e. four workshops each 
school year). Workshops will be designed to help evidence champions examine research and evidence 
relating to an agreed area of focus, and to develop, apply and evaluate school key stage-wide 
improvement strategies based on this evidence. 


Ensure between workshops that evidence champions are provided with intersessional tasks to guide 
their evidence sharing activity and to help them effectively implement their improvement strategies. 
Extensive support will also be provided to help schools with these tasks. 


Build capacity to ensure the sustainability of the approach, meaning that schools can continue to run 
RLCs and use evidence effectively, after the project ends. 


Structure of the Evaluation 


The evaluation is being conducted by the University of Bristol. Schools who agree to take part in the 
project have been randomly allocated to either the intervention group or a wait-list control group. This 
allocation happened in September 2014. 


The schools in the intervention group will receive support from LCLL, through RLCs in the 2014/2015, 
and 2015/2016 academic years. The schools in the control group will receive free access to RLCs from 
the 2016/17 academic year. In addition, those schools in the control group who participate fully in the 
evaluation will receive a financial incentive of £1000 at the end of the 2015/16 academic year. 


The evaluation data collection comprises three main strands: 


Pupil attainment at KS2 literacy and numeracy in 2015/16. These will be accessed using Unique Pupil 
Numbers provided by schools. The attainment of all pupils in this cohort in all the schools in the 
evaluation will be used. 


All teachers in all RLC and control schools in the evaluation will be asked to complete three short 
surveys about their awareness, understanding and use of research. These surveys will be collected in 
July 2014, June 2015, and June 2016. 


To understand how schools engage with RLCs, interviews will be conducted with selected school staff 
at two time points across the project in eight schools (six RLC schools and two control schools). 
Interviews will take between 30 and 60 minutes each. 


Use of Data 


Pupils’ KS2 scores and any other pupil data will be treated with the strictest confidence. Pupils’ KS2 
scores and other data will accessed through the National Pupil Database by the University of Bristol. 
This will be shared with LCLL, and with EEF. No individual school or pupil will be identified in any report 
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arising from the research. Fully anonymised data from the teacher surveys will be shared with the 
National Foundation for Educational Research, who have developed the survey instruments. 


Right to withdraw 


Although it is important for the evaluation to work with as full a dataset as possible, participation in the 
evaluation data collection is voluntary for teachers. You have the option not to take part in the study, 
and we will not share the names of those who decline to take part with their schools. You can choose 
to withdraw at any point in the study, for example when asked to take part survey, or if asked to take 
part in an interview. If you complete a survey and then change your mind about participation, you do 
not have to submit your responses. If you complete an interview and then change your mind about 
participation, please let us know by the end of July 2016. 


If you decide not take part in the evaluation, or any part thereof, please contact Jo Rose at the University 
of Bristol, with your name, class name/number, and school, using the contact details below. Any 
teachers that choose not to take part will not be identified to their schools. In the meantime, if you have 
any questions or queries please contact the evaluation team at the University of Bristol using the contact 
details below. 


We look forward to working with you over the coming two years. 


Dr Jo Rose jo.rose@bristol.ac.uk 
Professor Sally Thomas s.thomas@bristol.ac.uk 
Dr Tim Jay tim.jay@bristol.ac.uk 


CONSENT FORM FOR SCHOOL STAFF TAKING PART 
IN THE RESEARCH LEARNING COMMUNITIES PROJECT EVALUATION INTERVIEWS 
Conducted by researchers from University of Bristol 


| confirm that | have read and understood the information sheet relating to the Research Learning 
Communities project evaluation 


| volunteer to participate in an interview for the above named project. 


| have: Yes No 
Been given information explaining about the project Q O 
Had an opportunity to ask questions and discuss the project QO Q) 
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Received satisfactory answers to any questions I have asked UO O 


Received enough information to make a decision about participating | O 


I consent to my data being collected, stored anonymously, and used by the researcher for 
the purpose of the project (TICK as appropriate) 
Q) 


I agree to the interview being audio recorded, and the recording being used as an aid to 
data collection (TICK as appropriate): 
Q) 


I understand that I am free to withdraw my consent from participating in this project at 
any time until July 2016, without giving a reason (TICK as appropriate) : 
Q) 


Signed: Print name: Date: 


Having participated in the project evaluation: | give permission for the University of Bristol to keep 
and use the data | have provided during the course of the evaluation, for the purposes outlined in the 
information sheet. 


Signed: Print name: Date: 
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Appendix F: Parent opt-out consent 


A Seal Ele University of 
ndowmen § 
eae BRISTOL 


Dear Parent 


Teachers at (name of the school) are currently working with the Institute of Education at the University 
of London on a project called Research Learning Communities. Teachers from schools in the project 
are/will be working together to understand how they can develop and improve their teaching. The project 
is funded by the Education Endowment Foundation (EEF). 


The University of Bristol is evaluating the project. As part of the evaluation, we would like to use Key 
Stage Two SATs scores from children in Year 6 at your child’s school. This will help us understand 
whether the Research Learning Communities project has led to any change in pupil attainment. The 
SATs scores will be accessed from the National Pupil Database, using Unique Pupil Numbers from the 
school. The SATs scores will be anonymous, and the evaluation team from the University of Bristol will 
not be able to link your child’s scores to their name. 


These anonymous SATs scores will be linked with other data from the National Pupil Database (held 
by the Department for Education), and other official records, and shared for research purposes with the 
project team at the Institute of Education, the evaluation team at the University of Bristol, the EEF’s 
data archive, EEF’s data contractor FFT Education, and the UK Data Archive. The UK Data Archive 
can be accessed by other researchers for research purposes. Your child’s name or the name of any 
school will not be shared with the UK Data Archive. We will not use your child’s name or the name of 
the school in any report arising from the research. 


As your child will be doing Key Stage 2 SATs anyway, there should be no extra burden on your child’s 
time. However, if you prefer for your child’s SATs scores NOT to be used, please inform their teacher 
using the attached form, by 23 March 2016. There will be no consequences for your child if you choose 
to withdraw their data from this study. If you would like more information, please contact: 


Dr Sara Speybroeck, Research Associate, Graduate School of Education, University of Bristol. 
Email sara.speybroeck@bristol.ac.uk Telephone 0117 331 4159. 

Best wishes 

Dr Jo Rose, Lecturer in Education (Evaluation lead) 

Dr Sara Speybroeck, Research Associate 

Professor Sally Thomas, Professor of Education 


Graduate School of Education, University of Bristol 


Research Learning Communities Project Evaluation 
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If you would NOT like your child’s SATs scores to be used as part of the Research Learning 
Communities project evaluation, please complete the form below and return it to your child’s teacher. 


Name: of Parent/Carefisiic: ci saccs itelaccedodsae haw sh dae hecae Advet ide baw dadadder hice Madea 


Namesof Child). :c:cscccitctes ieieaudeereides toiGadsnitabineol abe maelatantded) ntdackalaias fuiduedalcad sass 


| would NOT like my child’s SATs scores to be used as part of the Research Learning Communities 
project evaluation. 
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Appendix G: Summary tables of sample sizes/missing data 


Table G.1: Analysis Samples 


School n/N Pupil n/N School n/N Pupil n/N 
(missing) (missing) (missing) (missing) 

Baseline 60/60 (0) 2619/2619(0) 59/59 (0) 2933/2933(0) 
KS2 outcome analysis 59/60 (1) 2402/2619(217 57/59 (2) 2564/2933(369 
Reading (Part 1) ) ) 
KS2 outcome analysis 59/60 (1) 2412/2619(207 57/59 (2) 2588/2933(345 
Numeracy (Part 1) ) ) 
KS2 outcome analysis 59/60 (1) 2412/2619(207 57/59 (2) 2585/2933(348 
GPS (Part 1) ) ) 
KS2 outcome analysis 47/60(13) 1432/2619(118 35/59(24) 1439/2933(149 
Reading (Part 2: teacher 7) 4) 
survey) 
KS2 outcome analysis 47/60(13) 1440/2619(117 35/59(24) 1446/2933(148 
Numeracy (Part 2: teacher 9) 7) 
survey) 
KS2 outcome analysis 47/60(13) 1440/2619(117 35/59(24) 1446/2933(148 
GPS (Part 2: teacher 9) 7) 
survey) 
KS2 outcome analysis 55/60(5) 2025/2619(594 49/59(10) 2106/2933(827 
Reading (Part 2: ) ) 
intervention dosage) 
KS2 outcome analysis 55/60(5) 2033/2619(586 49/59(10) 2119/2933(814 
Numeracy (Part 2: ) ) 
intervention dosage) 
KS2 outcome analysis 55/60(5) 2033/2619(586 49/59(10) 2120/2933(813 
GPS (Part 2: intervention ) ) 
dosage) 
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School n/N 


(missing) 


Table G.2 Analysis Samples of Teacher Survey Outcomes 


Teacher n/N 
(missing) 


School n/N 
(missing) 


Teacher n/N 
(missing) 
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Baseline 59/60 (0) 834/1037(0) 59/59 (0) 875/1102(0) 
Teacher Survey round 2 56/60(4) 441/1037(596) 52/59(7) 525/1102(577 
) 
Teacher Survey round 3 39/60(21) 235/1037(802) 40/59(19) 464/1102(638 
) 
Mean Range in Mean Range in 
percentage percentage of percentage of percentage 
of teacher teacher non teacher non of teacher 
non responses responses non 
responses across across responses 
across responded responded across 
responded schools schools responded 
schools schools 
Teacher Survey round 2 50.0% [0.1%, 92.3%] 47.8% [0.0%,95.2%] 
Teacher Survey round 3 61.2% [0.0%,93.8%] 42.9% [0.0%, 95.5%] 


Table G.3 Analysis Samples Pupil KS2 Outcomes Part 1: number of missing values for each 


variable in the sample (N=5552 from 119 schools) 


KS2 Reading 176 3.2% 
KS2 Maths 131 2.4% 
KS2 Grammar, Punctuation and 137 2.5% 
Spelling 

Standardised KS1 Reading 358 6.4% 
Standardised KS1 Writing 357 6.4% 
Standardised KS1 math 357 6.4% 
Standardised KS1 science 359 6.5% 
everFSM 2 0.0% 


Education Endowment Foundation 


68 


Research Learning Communities 


Notes: There were three schools missing in Part 1 analysis: School (3723003) in control group due to 
all 19 students missing KS2 reading score; School (8787044) in treatment group due to all 7 students 
missing KS2 reading score; School (8112042) in control group was dropped out due to no pupils in 
Year 5 and/or Year 6; 
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Table G.4 Analysis Samples Pupil KS2 Outcomes Part 2: number of missing values for each 
variable in the sample (n=4418 from 106 schools) 


With Unique ID provided for NPD dataset 4858 
Having the KS2 results recorded in the same school as they were in 4418 
Year 5 


KS2 Reading Score 118 2.7% 4,300 
KS2 Math Score 90 2.0% 4,328 

KS2 GPS Score 91 2.1% 4,327 
Year5 and Year 6 teachers 1,310 29.7% 3,108 
KS1reading score 177 4.0% 4,241 
KS1writing score 176 4.0% 4,242 
KS1maths score 176 4.0% 4,242 
KS1science score 177 4.0% 4,241 
everFSM 2 0.1% 4,416 


Table G.5: Comparison of school data for schools in Part 1 analysis (pupil outcomes) 
Variable Tal(=vavc-valdre)ame [cele e) Ofey alice) Mel celele) 
n/N n/N 


School-level (continuous) iaiceing Mean inane Mean 


Size of School 59/59 (0) aane 57/57 (0) 
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% FSM Eligibility 59/59 20.9% 57/57 (0) 22.4% 


( 
Attainment at KS2 59/59 ( Sree) 
(% achieving level 4 in reading, 80.4% 74.6% 
writing and maths) 

( 


0) 

0) 

% Pupils with English as an 59/59 (0) . 57/57 (0) " 

Additional Language (EAL) pm apie an 
n/N 


n/N Percentag 


Percentag 
(missing) (missing) 


EoYed aYolo) (N42) Mm (excl (cYe(o) ater 1p) 


Ofsted Rating 45/59 (14) 44/57 (13) 
Outstanding 9/59 (0) 11/57 (0) 
Good 35/59 (0) 31/57 (0) 
Requires Improvement 1/59 (0) 2/57 (0) 
Inadequate 0/59 (0) 0/57 (0) 
None 14/59 (0) 13/57 (0) 
School Location 

Urban 

Rural 

FSM eligibility 

High (>35%) 

Medium (20.1% - 35%) 


Low (< 20.1%) 


School Type 
Academy — Converter Mainstream 9/59 (0) 15% 5/57 (0) 9% 
Academy Sponsor Led Mainstream 5/59 (0) 8% 9/57 (0) 16% 
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Community 33/59 (0) 56% 29/57 (0) 51% 
Community Special 0/59 (0) 0% 1/57 (0) 2% 
Foundation 1/59 (0) 2% 1/57 (0) 2% 
Foundation Special 0/59 (0) 0% 0/57 (0) 0% 
Voluntary Aided 10/59 (0) 17% 8/57 (0) 14% 
Voluntary Controlled 1/59 (0) 2% 4/57 (0) 7% 


Table G.6: Comparison of school data for schools in Part 2 analysis (pupil outcomes linked to 
teacher data) 
Variable Tait-vavc-valdre)ame|aelel e) fey alige) Mel gele]e) 


School-level (continuous) ne He ae es 


Size of School 


% FSM Eligibility 


Attainment at KS2 


(% achieving level 4 in reading, 
writing and maths) 


% Pupils with English as an 
Additional Language (EAL) 


boTed aKoto) ia (=\"4=) Mm (ore L(cYele) a (er-l)) (missing) Beltbh ee sa ae 
Ofsted Rating 36/47 (11) 28/36 (8) 

Outstanding 7/47 (0) 15% 7/36 (0) 19% 
Good 29/47 (0) 62% 19/36 (0) 53% 
Requires Improvement 0/47 (0) 0% 2/36 (0) 6% 
Inadequate 0/47 (0) 0% 0/36 (0) 0% 
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None 11/47 (0) 23% 8/36 (0) 22% 
School Location 

Urban 

Rural 

FSM eligibility 

High (>35%) 

Medium (20.1% - 35%) 


Low (< 20.1%) 


School Type 


Academy — Converter Mainstream 
Academy Sponsor Led Mainstream 
Community 

Community Special 

Foundation 

Foundation Special 

Voluntary Aided 


Voluntary Controlled 


Note: The reduced sample (more conservative) reported here was used in models of KS2 outcome 
(KS2 Reading/ KS2 Numeracy/ KS2 GPS) with dosage, randomisation control variables, prior 
attainment and FSM and teacher background variables. 


Table G.7: Comparison of school data for schools in Teacher Survey (round 3) Outcomes 
Analysis 


MeeVee le) (=) Intervention group (Ofoy alice) Me cel] e) 


YAN YAN 


(missing) Mea (missing) ea 


School-level (continuous) 


Size of School 39/39 (0) 2873 40/40 (0) 362.8 
% FSM Eligibility 39/39 (0) 18.4% 40/40 (0) 19.0% 
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Attainment at KS2 oe 40/40 (0) 
(% achieving level 4 in reading, 80.1% 73.5% 
writing and maths) 


: me | 
% Pupils with English as an 
Additional Language (EAL) 39/39 (0) 20.7% 40/40 (0) 25.0% 
0 1 
y sn 


% Pupils with SEN 39/39 40/40 (0) 1.4% 
n/N 


n/N Percentag Percentag 
(missing) ) (missing) ) 


EsYed aYolo) (N42) Mm Cex-1(c\o(e) ater-|)) 


Ofsted Rating 
Outstanding 

Good 

Requires Improvement 
Inadequate 

None 

School Location 
Urban 

Rural 


FSM eligibility 


High (>35%) 


Medium (20.1% - 35%) 


Low (< 20.1%) 


School Type 

Academy — Converter Mainstream 5/39 (0) 13% 5/40 (0) 13% 
Academy Sponsor Led Mainstream 2/39 (0) 5% 6/40 (0) 15% 
Community 23/39 (0) 59% 19/40 (0) 48% 
Community Special 0/39 (0) 0% 0/40 (0) 0% 
Foundation 0/39 (0) 0% 1/40 (0) 3% 
Foundation Special 1/39 (0) 3% 0/40 (0) 0% 
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Voluntary Aided 7/39 (0) 18% 5/40 (0) 13% 


Voluntary Controlled 1/39 (0) 3% 4/40 (0) 10% 


Table G7.8:Table of Cross-tabulation (Status by Intervention/Control) 


Status (ereyaline)| Intervention 


group 
| h 
Classroom teacher 950 


Middle leader (e.g. head of department, 
subject or curriculum area leader, key s 


Senior leader (e.g. deputy or assistant 
headteacher) 


“ees pe fos fe 


[cael teacher 


ee ly/PPA cover teacher 


Table G7.9:Table of Cross-tabulation (Holds postgraduate qualifications by Intervention/Control) 


La Ko) (of=m ofey-yCele-(olUl-utcmelUr-linaler- LiLo) ary Control Intervention group 
group 


a ae a a oe 


Table G7.10:Table of Cross-tabulation (Year of teaching by Intervention/Control) 


Education Endowment Foundation 75 


Research Learning Communities 


group 


Years of teaching Oo} n) ice) Me] cele] o) Intervention Total 


20-29 years 
10-19 years 


First year of teaching (NQT) 


a 
ee 
ws 

eee 
—« 


Table G7.11:Table of Cross-tabulation (Year of teaching in current school by Intervention/Control) 


Years of teaching in current Oo) nh ice) Me] cele| ) Intervention ie) €| 
Yes aTofe)| group 


20-29 years 
10-19 years 


Less than 1 year 


oe 
ee 
we 
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Appendix H: Multilevel Modelling Results 


Table H.1 Multilevel model results for part 1 (outcome: KS2 Reading) 
(1) (2) (3) 


KS2 Reading KS2 Reading KS2 Reading 
Fixed Part: 
Treatment 0.02 (0.06) 0.02 (0.05) 
School FSM% -0.21~ (0.07) 0.03 (0.06) 
School average KS2 score 0.34" (0.07) 0.23" (0.06) 
school size small ref. ref. 
School size medium -0.13 (0.09) -0.10 (0.07) 
school size large -0.04 (0.09) -0.02 (0.08) 
Standardized KS1 Reading 0.35" (0.02) 
score 
Standardized KS1 Writing 0.14 (0.02) 
score 
Standardized KS1 Maths 0.15 (0.02) 
score 
Standardized KS1 Science 0.07" (0.02) 
score 
everFSM -0.13™ (0.02) 
Constant 0.02 (0.04) -0.06 (0.09) -0.08 (0.08) 
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Random Part: 

school: variance 0.13" (0.02) 0.08" (0.01) 0.06" (0.01) 
student: variance 0.86" (0.02) 0.86" (0.02) 0.47" (0.01) 
groups 116 116 116 

N 5286 5286 4966 

Deviance 14409.57 14363.35 10519.26 


Standard errors in parentheses 
“p<0.05,” p<0.01, p< 0.001 


Table H.2: Multilevel modeling results for part 1 (outcome: KS2 Numeracy) 


(1) (2) (3) 


KS2 Numeracy KS2 Numeracy KS2 Numeracy 

Fixed Part: 

Treatment 0.05 (0.07) 0.04 (0.06) 
School FSM% -0.11 (0.08) 0.15" (0.07) 
School average KS2 score 0.28" (0.08) 0.17" (0.07) 
school size small ref. ref. 

School size medium 0.08 (0.10) 0.06 (0.08) 
school size large 0.22" (0.10) 0.20° (0.09) 
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Standardized KS1 Reading 0.01 (0.02) 
score 

Standardized KS1 Writing 0.14" (0.02) 
score 

Standardized KS1 Maths 0.56" (0.02) 
score 

Standardized KS1 Science 0.03" (0.01) 
score 

everFSM -0.12™ (0.02) 
Constant -0.02 (0.04) -0.29" (0.10) -0.30" (0.09) 
Random Part: 

School: variance 0.14" (0.02) 0.11" (0.02) 0.10" (0.01) 
Student: variance 0.86" (0.02) 0.86" (0.02) 0.42" (0.01) 
groups 116 116 116 

N 5331 5331 5000 

Deviance 14559.13 14537.72 10141.43 


Standard errors in parentheses 
“p<0.05,™” p<0.01, p< 0.001 
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(1) 


(2) 


(3) 


KS2 GPS KS2 GPS KS2 GPS 
Fixed Part: 
Treatment 0.07 (0.07) 0.07 (0.06) 
School FSM% -0.19° (0.08) 0.05 (0.07) 
School average KS2 score 0.22” (0.08) 0.07 (0.08) 
school size small ref. ref. 
schoolsize==medium 0.11 (0.10) 0.16 (0.09) 
school size large 0.23° (0.10) 0.28" (0.09) 
Standardized KS1 Reading 0.33" (0.02) 
score 
Standardized KS1 Writing 0.30" (0.02) 
score 
Standardized KS1 Maths 0.16" (0.02) 
score 
Standardized KS1 Science 0.02 (0.01) 
score 
everFSM -0.10™ (0.02) 
Constant -0.02 (0.04) -0.25° (0.11) -0.28" (0.09) 
Random Part: 
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School: variance 0.15 (0.02) 0.12” (0.02) 0.11 (0.02) 
Student: variance 0.85" (0.02) 0.85" (0.02) 0.35" (0.01) 
groups 116 116 116 

N 5325 5325 4997 

Deviance 14477.07 14455.94 9233.19 


Standard errors in parentheses 
“p<0.05, ~p<0.01, p< 0.001 
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Table H.4 Multilevel modeling results for part 1 (interaction: Treatment “everFSM) 


(1) (2) (3) 


Outcome KS2 Reading KS2 Numeracy KS2 GPS 

Fixed Part: 

Standardized KS1 Reading 0.35" (0.02) 0.01 (0.02) 0.33" (0.02) 
score 

Standardized KS1 Writing 0.14" (0.02) 0.14" (0.02) 0.30" (0.02) 
score 

Standardized KS1 Maths 0.15 (0.02) 0.56" (0.02) 0.16" (0.02) 
score 

Standardized KS1 Science 0.07" (0.02) 0.03" (0.01) 0.02 (0.01) 
score 

School FSM% 0.03 (0.06) 0.15° (0.07) 0.05 (0.07) 
School average KS2 score 0.23" (0.06) 0.177 (0.07) 0.07 (0.08) 
school size small ref. ref. ref. 

School size medium -0.10 (0.07) 0.06 (0.08) 0.15 (0.09) 
school size large -0.02 (0.08) 0.20° (0.09) 0.28” (0.09) 
Treatment 0.01 (0.06) 0.04 (0.06) 0.05 (0.07) 
everFSM -0.15° (0.03) -0.11™ (0.03) -0.13 (0.03) 
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Treatment X everFSM 0.03 (0.05) -0.02 (0.04) 0.06 (0.04) 
Constant -0.08 (0.08) -0.30" (0.09) -0.26" (0.09) 
Random Part: 

School: variance 0.06" (0.01) 0.10" (0.01) 0.10" (0.02) 
Student: variance 0.47™" (0.01) 0.42" (0.01) 0.35" (0.01) 
groups 116 116 116 

N 4966 5000 4997 

Deviance 10518.84 10141.26 9230.95 


Standard errors in parentheses 
“p<0.05,” p<0.01, ™ p< 0.001 
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Table H.5 multilevel modeling results for part 1 (Schools with literacy or numeracy focus vs Control group) 


(1) (2) (3) 


KS2 Reading KS2 Numeracy KS2 GPS 
Fixed Part: 
Standardized KS1 Reading 0.35" (0.02) 0.01 (0.02) 0.33" (0.02) 
score 
Standardized KS1 Writing 0.14 (0.02) 0.14 (0.02) 0.30" (0.02) 
score 
Standardized KS1 Maths 0.15" (0.02) 0.56" (0.02) 0.16" (0.02) 
score 
Standardized KS1 Science 0.07" (0.02) 0.03" (0.01) 0.02 (0.01) 
score 
School FSM% 0.03 (0.06) 0.15° (0.07) 0.05 (0.07) 
School average KS2 score 0.22" (0.06) 0.18" (0.08) 0.07 (0.08) 
school size small ref. ref. ref. 
Schoolsize medium -0.10 (0.07) 0.07 (0.09) 0.16 (0.09) 
school size large -0.02 (0.08) 0.21° (0.09) 0.28" (0.09) 
everFSM -0.13" (0.02) -0.12 (0.02) -0.10™ (0.02) 
Project Focus Literacy 0.02 (0.06) 0.03 (0.07) 0.06 (0.07) 
Project Focus Numeracy -0.01 (0.11) 0.09 (0.13) 0.11 (0.13) 
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Constant -0.08 (0.08) p41” (0.09) -0.28" (0.10) 
Random Part: 

School: variance 0.06 (0.01) 0.10 (0.01) 0.11™ (0.02) 
Student: variance 0.47" (0.01) 0.42 (0.01) 0.35" (0.01) 
groups 116 116 116 

N 4966 5000 4997 

Deviance 10519.16 10141.17 9233.08 


Standard errors in parentheses 
“p<0.05,~ p<0.01, p< 0.001 
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Table H.6 Multilevel modeling results for part 2 (teacher survey measures in use analysis, outcome: KS2 Reading Score) 


(1) 


(2) 


(3) 


(4) 


(5) 


Fixed Part: 


Standardized measure1 score 


Standardized measure2 score 


Standardized measure3 score 


Standardized measure4 score 


Standardized measure5 score 


Standardized measure6 score 


everFSM 


Standardized KS1 Reading score 


Standardized KS1 Writing score 


Standardized KS1 Science score 
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-0.00 
(0.02) 


eOuie. 
(0.03) 
Gage 
(0.03) 
0.15%" 
(0.03) 
6077 
(0.02) 


0.04 
(0.02) 


stg 
(0.03) 
oda" 
(0.03) 
0.15" 
(0.03) 
0077 
(0.02) 


0.01 
(0.02) 


-0.09" 


(0.03) 
0.00 
(0.02) 


0.17" 


(0.02) 
0.01 
(0.02) 


-0.01 
(0.02) 


-0.09" 
(0.03) 
0.00 
(0.02) 
an Waa 
(0.02) 
0.01 
(0.02) 


0.02 
(0.02) 


-0.09" 


(0.03) 
0.00 
(0.02) 


0.17" 


(0.02) 
0.01 
(0.02) 


-0.00 
(0.02) 


-0.09" 


(0.03) 
0.00 
(0.02) 


0.17" 


(0.02) 
0.01 
(0.02) 
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Standardized KS1 Maths score 0.177 0.17” 0.57” 0.57" 0.57" 0.57" 0.57" 
(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) 
Constant 0.04 0.04 0.03 0.04 0.04 0.04 0.04 
(0.03) (0.03) (0.04) (0.04) (0.04) (0.04) (0.04) 
Random Part: 
School: variance 0.07" 0.07" 0.11" 0.11" 0.11" 0.11" 0.11" 
(0.01) (0.01) (0.02) (0.02) (0.02) (0.02) (0.02) 
Student: variance 0.45" 0.45" 0.42" 0.42" 0.42" 0.42" 0.42" 
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
groups 82 82 82 82 82 82 82 
N 2871 2871 2886 2886 2886 2886 2886 
Deviance 5997.28 5994.54 5851.98 5851.94 5851.52 5852.02 5842.25 


Standard errors in parentheses 
“p<0.05,” p<0.01, ™ p< 0.001 
Table H.7 Multilevel modeling results for part 2 (teacher survey measures in use analysis, outcome: KS2 Numeracy Score) 


(1) (2) (3) (4) (5) (6) (7) 


Fixed Part: 
Standardized measure1 score 0.04 0.03 
(0.02) (0.03) 
Standardized measure2 score 0.06" 0.06" 
(0.02) (0.03) 
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Standardized measure4 score 


Standardized measure5 score 


Standardized measure6 score 


everFSM 


Standardized KS1 Reading score 


Standardized KS1 Writing score 


Standardized KS1 Science score 


Standardized KS1 Maths score 


Constant 


Random Part: 


School: variance 
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-0.09" 
(0.03) 
0.002 
(0.02) 
oa 
(0.02) 
0.01 
(0.02) 
0.57" 
(0.02) 
0.04 
(0.04) 


0.117 


-0.09" 
(0.03) 
0.002 
(0.02) 
0.16" 
(0.02) 
0.01 
(0.02) 
0.57" 
(0.02) 
0.03 
(0.04) 


0.11™ 


0.01 
(0.02) 


-0.09" 


(0.03) 
0.002 
(0.02) 


0.17" 


(0.02) 
0.01 
(0.02) 


0.57" 


(0.02) 
0.03 
(0.04) 


0.11™ 


-0.01 
(0.02) 


-0.09" 
(0.03) 
0.002 
(0.02) 
0.17" 
(0.02) 
0.01 
(0.02) 
0.577" 
(0.02) 
0.04 
(0.04) 


0.117 
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0.02 
(0.02) 


-0.09" 


(0.03) 
0.002 
(0.02) 


0.17" 


(0.02) 
0.01 
(0.02) 


0.57" 


(0.02) 
0.04 
(0.04) 


0.117 


-0.00 
(0.02) 


-0.09" 


(0.03) 
0.002 
(0.02) 


0.17” 


(0.02) 
0.01 
(0.02) 


057" 


(0.02) 
0.04 
(0.04) 


0.117 


0.03 
(0.03) 
-0.00 
(0.03) 
0.00 
(0.02) 
-0.02 
(0.02) 


-0.09" 


(0.03) 
0.002 
(0.02) 


0.16" 


(0.02) 
0.01 
(0.02) 


057° 


(0.02) 
0.04 
(0.04) 


0.117 
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(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) 
Student: variance 0.42” 0.42” 0.42” 0.42” 0.42 0.42” 0.42" 
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
groups 82 82 82 82 82 82 82 
N 2886 2886 2886 2886 2886 2886 2886 
Deviance 5848.69 5845.67 5851.98 5851.93 5851.51 5852.02 5842.25 


Standard errors in parentheses 
“p<0.05,™ p< 0.01, p< 0.001 


Table H.8 Multilevel modeling results for part 2 (teacher survey measures in use analysis ,outcome: KS2 GPS Score ) 


(1) (2) (3) (4) (5) (6) (7) 


Fixed Part: 
Standardized measure1 score 0.05° 0.04 
(0.02) (0.03) 
Standardized measure2 score 0.06" 0.06" 
(0.02) (0.03) 
Standardized measure3 score 0.01 0.03 
(0.02) (0.03) 
Standardized measure4 score 0.01 0.02 
(0.02) (0.02) 
Standardized measured score 0.03 0.03 
(0.02) (0.02) 
Standardized measure6 score -0.01 -0.04 
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(0.02) (0.02) 
everFSM -0.09" -0.09" -0.09" -0.09" -0.08" -0.09" -0.09" 
(0.03) (0.03) (0.03) (0.03) (0.03) (0.03) (0.03) 
Standardized KS1 Reading score 0.32" 0.32" 0.32" 0.32" 0.32" 0.32" 0.32" 
(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) 
Standardized KS1 Writing score 0.31" 0.31" 0.31" 0.31" 0.31" 0.31" 0.31" 
(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) 
Standardized KS1 Science score -0.00 -0.01 -0.00 -0.01 -0.01 -0.01 -0.00 
(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) 
Standardized KS1 Maths score 0.19" 0.19” 0.18” 0.18" 0.18" 0.18” 0.18" 
(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) 
Constant 0.03 0.03 0.03 0.03 0.03 0.03 0.04 
(0.04) (0.04) (0.04) (0.04) (0.04) (0.04) (0.04) 
Random Part: 
School: variance 0.11" 0.11" 0.10" 0.10" 0.11" 0.10" 0.11" 
(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) 
Student: variance 0.35 0.35" 0.35" 0.35" 0.35" 0.35" 0.35" 
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
groups 82 82 82 82 82 82 82 
N 2886 2886 2886 2886 2886 2886 2886 
Deviance 5347.02 5345.86 5351.85 5352.08 5349.81 5352.08 5336.46 
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Standard errors in parentheses 
“p<0.05, ™ p<0.01, p< 0.001 
Table H.9 Multilevel modeling results for part 2 (dosage analysis) 


(1) (2) (3) 


KS2 Reading Score KS2 GPS Score KS2 Numeracy 
Score 

Fixed Part: 
centered workshop attendance 0.01 (0.00) 0.01 (0.00) 0.01 (0.01) 
(treatment dosage) 
School FSM% -0.02 (0.06) -0.02 (0.07) 0.12 (0.07) 
School average KS2 score 0.18" (0.07) 0.05 (0.07) 0.15° (0.08) 
School size small ref ref ref 
School size medium -0.06 (0.08) 0.21" (0.09) 0.05 (0.09) 
School size large -0.02 (0.09) 0.33" (0.09) 0.16 (0.10) 
everFSM -0.11" (0.03) -0.09"" (0.02) -0.10°" (0.02) 
Standardized KS1 Reading score 0.35" (0.02) 0.33" (0.02) 0.01 (0.02) 
Standardized KS1 Writing score 0.15 (0.02) 0.32” (0.02) 0.16" (0.02) 
Standardized KS1 Science score 0.077 (0.02) -0.00 (0.01) 0.02 (0.02) 
Standardized KS1 Maths score 0.16™ (0.02) 0.17" (0.02) 0.56™ (0.02) 
Constant -0.04 (0.08) -0.26" (0.09) -0.23° (0.09) 
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Random Part: 

School: variance 0.06" (0.01) 0.09" (0.01) 0.09" (0.02) 
Student: variance 0.48" (0.01) 0.35" (0.01) 0.42” (0.01) 
groups 104 104 104 

N 4131 4153 4152 

Deviance 8857.63 7705.61 8434.25 


Standard errors in parentheses 
“p<0.05, ™ p<0.01, p< 0.001 


Table H.10 Multilevel modeling results for part 2 (teacher outcome analysis for measure 1: Positive disposition to academic research in informing teaching 


practice) 
(1) (2) (3) 
Standardized Standardized Standardized 
measure1 score measure score measure score 

Fixed Part: 
Treatment 0.21" (0.10) 0.14 (0.09) 
School FSM% -0.08 (0.10) -0.06 (0.10) 
School average KS2 score 0.16 (0.11) 0.14 (0.10) 
school size small ref. ref. 
school size medium -0.03 (0.13) 0.01 (0.13) 
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school size large -0.04 (0.14) 0.04 (0.13) 
Classroom teacher ref. 
Middle leader (e.g. head of department, subject or 0.34" (0.09) 


curriculum area leader) 


Senior leader (e.g. deputy or assistant headteacher) 0.58" (0.13) 
Headteacher, principal or director 1.10" (0.21) 
SEN teacher 0.39 (0.42) 
Supply, cover or PPA teacher 0.15 (0.25) 
High level teaching assistant 0.86 (0.47) 
Music or PE extra curricula teacher -0.07 (0.47) 
with postgraduate qualifications 0.47 (0.13) 
year of teaching 0.02 (0.04) 
year of teaching in the school -0.03 (0.04) 
Constant 0.00 (0.05) -0.12 (0.15) -0.48° (0.23) 
Random Part: 

School: variance 0.05" (0.03) 0.03" (0.02) 0.03" (0.02) 
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Student: variance 0.94 (0.05) 0.95 (0.05) 0.84" (0.05) 
groups 79 79 79 

N 699 699 699 

Deviance 1974.12 1963.08 1884.70 


Standard errors in parentheses 
“p<0.05, ~p<0.01, p< 0.001 


Table H.11 Multilevel modeling results for part 2 (teacher outcome analysis for measure 2: Use of academic research to inform selection of teaching 
approaches) 


(1) (2) (3) 


Standardized Standardized Standardized 
measure2 score measure2 measure2 
score score 

Fixed Part: 

Treatment 0.20° (0.10) 0.13 (0.10) 
School FSM% -0.04 (0.11) 0.02 (0.10) 
School average KS2 score 0.12 (0.11) 0.12 (0.11) 
school size small ref. ref. 

school size medium -0.05 (0.14) 0.01 (0.13) 
school size large -0.14 (0.14) -0.08 (0.14) 
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Classroom teacher ref. 

Middle leader (e.g. head of department, subject or 0.22” (0.09) 
curriculum area leader) 

Senior leader (e.g. deputy or assistant headteacher) 0.25 (0.13) 
Headteacher, principal or director 1.25" (0.21) 
SEN teacher 0.46 (0.43) 
Supply, cover or PPA teacher 0.19 (0.25) 
High level teaching assistant 0.67 (0.47) 
Music or PE extra curricula teacher -0.50 (0.47) 
with postgraduate qualifications 0.18 (0.13) 
year of teaching 0.07 (0.04) 
year of teaching in the school 0.00 (0.04) 
Constant 0.01 (0.05) -0.06 (0.15) 0.02 (0.24) 
Random Part: 

School: variance 0.06" (0.03) 0.04" (0.02) 0.04" (0.02) 
Student: variance 0.94 (0.05) 0.94 (0.05) 0.84" (0.05) 
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groups 
N 


Deviance 


79 79 
699 699 
1972.93 1963.66 


79 
699 


1888.56 


Standard errors in parentheses 
“p<0.05,” p<0.01, p< 0.001 


Table H.12 Multilevel modeling results for part 2 (teacher outcome analysis for measure 3: Perception that academic research is not useful to teaching) 


(1) (2) 


(3) 


Standardized Standardized Standardized 
measure3 measure3 measure3 
score score score 

Fixed Part: 

Treatment -0.17 (0.09) -0.13 (0.09) 
School FSM% 0.16 (0.10) 0.11 (0.10) 
School average KS2 score -0.02 (0.10) -0.03 (0.10) 
school size small ref. ref. 

school size medium -0.01 (0.13) -0.04 (0.13) 
school size large -0.12 (0.13) -0.16 (0.13) 


Classroom teacher 
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Middle leader (e.g. head of department, subject or curriculum -0.22° (0.09) 
area leader) 

Senior leader (e.g. deputy or assistant headteacher) -0.41~ (0.13) 
Headteacher, principal or director -0.95"" (0.21) 
SEN teacher 0.51 (0.44) 
Supply, cover or PPA teacher 0.114 (0.26) 
High level teaching assistant -0.55 (0.49) 
Music or PE extra curricula teacher -0.27 (0.48) 
with postgraduate qualifications -0.04 (0.13) 
year of teaching -0.05 (0.04) 
year of teaching in the school 0.02 (0.05) 
Constant -0.01 (0.04) 0.06 (0.14) 0.15 (0.24) 
Random Part: 

School: variance 0.03" (0.02) 0.02" (0.02) 0.02" (0.02) 
Student: variance 0.97 (0.05) 0.97 (0.05) 0.91 (0.05) 
groups 79 79 79 
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N 


Deviance 


699 699 


1979.76 1972.03 
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699 


1927.96 


Standard errors in parentheses 
“p<0.05, ~p<0.01, p< 0.001 


Table H.13 Multilevel modeling results for part 2 (teacher outcome analysis for measure 4: Perception that own school does not encourage use of academic 


research) 
(1) (2) (3) 
Standardized Standardized Standardized 
measure4 measure4 measure4 
score score score 

Fixed Part: 
Treatment -0.11 (0.11) -0.07 (0.11) 
School FSM% 0.22 (0.12) 0.16 (0.12) 
School average KS2 score -0.05 (0.12) -0.07 (0.12) 
school size small ref. ref. 
school size medium 0.23 (0.14) 0.22 (0.14) 
school size large 0.06 (0.15) 0.08 (0.15) 


Classroom teacher 
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Middle leader (e.g. head of department, subject or curriculum -0.04 (0.09) 
area leader) 

Senior leader (e.g. deputy or assistant headteacher) -0.25 (0.13) 
Headteacher, principal or director -0.53° (0.21) 
SEN teacher 0.30 (0.43) 
Supply, cover or PPA teacher 0.16 (0.26) 
High level teaching assistant -0.57 (0.48) 
Music or PE extra curricula teacher -0.23 (0.48) 
with postgraduate qualifications 0.13 (0.13) 
year of teaching -0.01 (0.04) 
year of teaching in the school -0.10° (0.05) 
Constant 0.00 (0.05) -0.14 (0.16) -0.54° (0.25) 
Random Part: 

School: variance 0.10 (0.03) 0.07" (0.03) 0.07" (0.03) 
Student: variance 0.90 (0.05) 0.90 (0.05) 0.87" (0.05) 
groups 79 79 79 
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N 699 699 699 
Deviance 1961.21 1950.85 1923.88 


Standard errors in parentheses 
“p<0.05, ~p<0.01, p< 0.001 


Table H.14 Multilevel modeling results for part 2 (teacher outcome analysis for measure 5: Active engagement with online evidence platforms) 


(1) (2) (3) 


Standardized Standardized Standardized 
measure5 measured measure5 
score score score 
Fixed Part: 
Treatment 0.12 (0.09) 0.06 (0.09) 
School FSM% -0.14 (0.10) -0.10 (0.10) 
School average KS2 score -0.07 (0.10) -0.07 (0.10) 
school size small ref. ref. 
school size medium -0.06 (0.13) -0.05 (0.12) 
school size large -0.05 (0.13) -0.04 (0.13) 
Classroom teacher ref. 
Middle leader (e.g. head of department, subject or curriculum 0.20° (0.09) 
area leader) 
Senior leader (e.g. deputy or assistant headteacher) 0.54” (0.13) 
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Headteacher, principal or director 1.167 (0.21) 
SEN teacher 0.81 (0.44) 
Supply, cover or PPA teacher -0.29 (0.26) 
High level teaching assistant 0.26 (0.48) 
Music or PE extra curricula teacher -0.80 (0.48) 
with postgraduate qualifications -0.03 (0.13) 
year of teaching -0.04 (0.04) 
year of teaching in the school 0.06 (0.05) 
Constant -0.00 (0.04) 0.10 (0.14) 0.03 (0.24) 
Random Part: 

School: variance 0.02" (0.02) 0.02” (0.02) 0.02" (0.02) 
Student: variance 0.98 (0.06) 0.98 (0.05) 0.90 (0.05) 
groups 79 79 79 

N 699 699 699 

Deviance 1981.28 1976.96 1920.89 


Standard errors in parentheses 
“p<0.05, ™p<0.01, p< 0.001 


Education Endowment Foundation 101 


Research Learning Communities 


Table H.15 Multilevel modeling results for part 2 (teacher outcome analysis for measure 6: Knowledge of research findings and methods) 


(1) (2) (3) 


Standardized Standardized Standardized 
measure6 measure6 measure6 
score score score 
Fixed Part: 
Treatment 0.00 (0.10) -0.09 (0.09) 
School FSM% -0.17 (0.11) -0.10 (0.10) 
School average KS2 score 0.00 (0.11) -0.01 (0.10) 
school size small ref. ref. 
school size medium -0.11 (0.13) -0.06 (0.13) 
school size large -0.18 (0.14) -0.12 (0.13) 
Classroom teacher ref. 
Middle leader (e.g. head of department, subject or curriculum 0.05 (0.08) 
area leaderi“%o 
Senior leader (e.g. deputy or assistant headteacher) 0.41™ (0.13) 
Headteacher, principal or director 1.28" (0.20) 
SEN teacher 0.19 (0.42) 
Supply, cover or PPA teacher -0.04 (0.25) 
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High level teaching assistant -0.05 (0.47) 
Music or PE extra curricula teacher -0.98° (0.47) 
with postgraduate qualifications 0.39” (0.13) 
year of teaching 0.00 (0.04) 
year of teaching in the school -0.09° (0.04) 
Constant 0.01 (0.05) 0.20 (0.15) 0.43 (0.23) 
School: variance 0.05" (0.02) 0.03" (0.02) 0.03" (0.02) 
Student: variance 0.95 (0.05) 0.95 (0.05) 0.84" (0.05) 
groups 79 79 79 

N 699 699 699 

Deviance 1976.47 1970.69 1881.36 


Standard errors in parentheses 
“p<0.05, ~ p<0.01, p< 0.001 
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Appendix |: Multilevel modelling syntax code 


Al the syntax codes below were used in software Stata 14 (StataCorp, 2015)®. 


//Part 1 
//KS2 reading score 
eststo: xtmixed stdKS2_READSCORE || LAESTAB:, variance 


eststo: xtmixed stdKS2 READSCORE treatment FSMsper KS2s_ schoolsize2 schoolsize3 || 
LAESTAB:,mle variance 


eststo: xtmixed stdKS2_READSCORE treatment zKS1reading zKS1writing zKS1math zKS1science 
EVERFSM_ALL_SPR16 FSMsper KS2s schoolsize2 schoolsize3 || LAESTAB:,mle variance 


/ KKK KK KKK KK EKER KKK RRR KK KKK ER KERR ERE KK ERK KKK ERR ER ERE EER ERR KEERE 


eststo clear 
//KS2 Maths score 
eststo: xtmixed stdKS2_MATSCORE || LAESTAB:, variance 


eststo: xtmixed stdKS2_MATSCORE treatment FSMsper KS2s_ schoolsize2 schoolsize3 || 
LAESTAB:,mle variance 


eststo: xtmixed stdKS2_MATSCORE treatment zKS1reading zKS1writing zKS1math zKS1science 
EVERFSM_ALL_SPR16 FSMsper KS2s schoolsize2 schoolsize3 || LAESTAB:,mle variance 


if BERERAAEREBSS 2 ASAE EEREA SESSA SSE SEER ERES SHAE ESE EEE REBEL ERA RARE 


eststo clear 
//KS2 GPS score 
eststo: xtmixed stdKS2_GPSSCORE || LAESTAB:, variance 


eststo: xtmixed stdKS2_GPSSCORE treatment FSMsper KS2s_ schoolsize2 schoolsize3 || 
LAESTAB:,mle variance 


eststo: xtmixed stdKS2_GPSSCORE treatment zKStreading zKS1writing zZKS1math zKS1science 
EVERFSM_ALL_SPR16 FSMsper KS2s schoolsize2 schoolsize3 || LAESTAB:,mle variance 


8 StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP. 


Education Endowment Foundation 104 


Research Learning Communities 


//FSM*treatment 


eststo: xtmixed stdKS2_READSCORE zKS treading zKS1writing ZKS1math zKSiscience FSMsper 
KS2s schoolsize2 schoolsize3 treatment##EVERFSM_ALL_SPR16|| LAESTAB:,mle variance 


eststo: xtmixed stdKS2_MATSCORE zKStreading zKS1writing zKSimath zKS1science FSMsper 
KS2s schoolsize2 schoolsize3 treatment##EVERFSM_ALL_SPR16|| LAESTAB:,mle variance 


eststo: xtmixed stdKS2_GPSSCORE zKStreading zKS1writing zZKSimath zKS1science FSMsper 
KS2s schoolsize2 schoolsize3 treatment##EVERFSM_ALL_SPR16|| LAESTAB:,mle variance 


//literacy and numeracy focus 


eststo: xtmixed stdKS2_READSCORE zKS treading zKS1writing zKS1math zKSi1science FSMsper 
KS2s_ schoolsize2 schoolsize3 EVERFSM_ALL_SPR16 literacyProFocus numberacyProFocus]| 
LAESTAB:,mle variance 


eststo: xtmixed stdKS2_MATSCORE zKStreading zKS1writing zZKS1math zKSiscience FSMsper 
KS2s_ schoolsize2 schoolsize3 EVERFSM_ALL_SPR16 literacyProFocus numberacyProFocus]| 
LAESTAB:,mle variance 


eststo: xtmixed stdKS2_GPSSCORE zKStreading zKS1writing zKSimath zKS1science FSMsper 
KS2s_ schoolsize2 schoolsize3 EVERFSM_ALL_SPR16 literacyProFocus numberacyProFocus]| 
LAESTAB:,mle variance 


//Part2 
//KS2 outcome 


eststo:xtmixed stdKS2_READSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef1|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_READSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef2|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_READSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef3|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_READSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef4|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_READSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef5|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_READSCORE zKSireading zKS1writing zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef6|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_READSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16_ stdaveragef1 stdaveragef2 stdaveragef3 stdaveragef4 stdaveragef5 
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stdaveragef6|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_MATSCORE zKSireading zKSiwriting ZzKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef1|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_MATSCORE zKSireading zKSiwriting ZzKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef2|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_MATSCORE zKSireading zKSiwriting ZzKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef3|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_MATSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef4|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_MATSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef5|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_MATSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef6|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_MATSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16_ stdaveragef1 stdaveragef2 stdaveragef3 stdaveragef4 stdaveragef5 
stdaveragef6|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_GPSSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR(16 stdaveragef1|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_GPSSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef2|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_GPSSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef3|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_GPSSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef4|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_GPSSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef5|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_GPSSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16 stdaveragef6]|| y6t2_URN:, variance 


eststo:xtmixed stdKS2_GPSSCORE zKSireading zKSiwriting zKSimath zKS1science 
EVERFSM_ALL_SPR16_ stdaveragef1 stdaveragef2 stdaveragef3 stdaveragef4 stdaveragef5 
stdaveragef6|| y6t2_URN:, variance 


xtmixed stdKS2 READSCORE  centered_attendence16 FSMsper KS2s _ schoolsizenew2 
schoolsizenew3 zKSireading zKS1writing zZKSimath zKSiscience EVERFSM_ALL_SPR16 || 
y6t2_URN:, variance 


xtmixed stdKS2_GPSSCORE  centered_attendence16 FSMsper KS2s _ schoolsizenew2 
schoolsizenew3 zKSireading zKS1writing zZKSimath zKSiscience EVERFSM_ALL_SPR16 || 
y6t2_URN:, variance 


xtmixed stdKS2_MATSCORE  centered_attendencei6 FSMsper KS2s _ schoolsizenew2 
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schoolsizenew3 zKSireading zKS1writing zZKSimath zKSiscience EVERFSM_ALL_SPR16 || 
y6t2_URN:, variance 


//teacher outcome 
xtmixed stdmeasure1 ||URN:,mle variance 
xtmixed stdmeasure1 treatment FSMsper KS2s schoolsizenew2 schoolsizenew3||URN:,mle variance 


xtmixed stdmeasure1 treatment FSMsper KS2s schoolsizenew2 schoolsizenew3 role82 role83 role84 
role85 role86 role87 role88 qulification Q6 Q7||URN:,mle variance 


xtmixed stdmeasure2||URN:,mle variance 
xtmixed stdmeasure2 treatment FSMsper KS2s schoolsizenew2 schoolsizenew3||URN:,mle variance 


xtmixed stdmeasure2 treatment FSMsper KS2s schoolsizenew2 schoolsizenew3 role82 role83 role84 
role85 role86 role87 role88 qulification Q6 Q7||URN:,mle variance 


xtmixed stdmeasure3||URN:,mle variance 
xtmixed stdmeasure3 treatment FSMsper KS2s schoolsizenew2 schoolsizenew3||URN:,mle variance 


xtmixed stdmeasure3 treatment FSMsper KS2s schoolsizenew2 schoolsizenews role82 role83 role84 
role85 role86 role87 role88 qulification Q6 Q7||URN:,mle variance 


xtmixed stdmeasure4||URN:,mle variance 
xtmixed stdmeasure4 treatment FSMsper KS2s schoolsizenew2 schoolsizenew3||URN:,mle variance 


xtmixed stdmeasure4 treatment FSMsper KS2s schoolsizenew2 schoolsizenew3 role82 role83 role84 
role85 role86 role87 role88 qulification Q6 Q7||URN:,mle variance 


xtmixed stdmeasure5||URN:,mle variance 
xtmixed stdmeasure5 treatment FSMsper KS2s schoolsizenew2 schoolsizenew3||URN:,mle variance 


xtmixed stdmeasure5 treatment FSMsper KS2s schoolsizenew2 schoolsizenews role82 role83 role84 
role85 role86 role87 role88 qulification Q6 Q7||URN:,mle variance 


xtmixed stdmeasure6||URN:,mle variance 
xtmixed stdmeasure6 treatment FSMsper KS2s schoolsizenew2 schoolsizenew3||URN:,mle variance 


xtmixed stdmeasure6 treatment FSMsper KS2s schoolsizenew2 schoolsizenews role82 role83 role84 
role85 role86 role87 role88 qulification Q6 Q7||URN:,mle variance 
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//anova 


anova meanminew schoolsizenewnew2_ schoolsizenewnew3 
treat#round[aweight=freqnew] if school_pickone== 


anova meanm2new schoolsizenewnew2_ schoolsizenewnew3 
treat#round[aweight=freqnew] if school_pickone== 


anova meanm3new schoolsizenewnew2_ schoolsizenewnew3 
treat#round[aweight=freqnew] if school_pickone== 


anova meanm4new schoolsizenewnew2_ schoolsizenewnew3 
treat#round[aweight=freqnew] if school_pickone== 


anova meanmdnew schoolsizenewnew2_ schoolsizenewnew3 
treat#round[aweight=freqnew] if school_pickone== 


anova meanm6new schoolsizenewnew2_ schoolsizenewnew3 
treat#round[aweight=freqnew] if school_pickone== 
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